Re: [gentoo-user] replacement for ftp?
Alan McKinnon writes: > On 03/05/2017 22:04, lee wrote: >> Alan McKinnon writes: >> >>> On 30/04/2017 03:11, lee wrote: "Poison BL." writes: > On Sat, Apr 29, 2017 at 3:24 PM, lee wrote: > >> Mick writes: >> >>> On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote: On 25/04/2017 16:29, lee wrote: > Hi, > > since the usage of FTP seems to be declining, what is a replacement > which is at least as good as FTP? > > I'm aware that there's webdav, but that's very awkward to use and > missing features. Why not stick with ftp? Or, put another way, why do you feel you need to use something else? There's always dropbox >>> >>> >>> Invariably all web hosting ISPs offer ftp(s) for file upload/download. >> If you >>> pay a bit more you should be able to get ssh/scp/sftp too. Indeed, many >> ISPs >>> throw in scp/sftp access as part of their basic package. >>> >>> Webdav(s) offers the same basic upload/download functionality, so I am >> not >>> sure what you find awkward about it, although I'd rather use lftp >> instead of >>> cadaver any day. ;-) >>> >>> As Alan mentioned, with JavaScript'ed web pages these days there are >>> many >>> webapp'ed ISP offerings like Dropbox and friends. >>> >>> What is the use case you have in mind? >> >> transferring large amounts of data and automatization in processing at >> least some of it, without involving a 3rd party >> >> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB, >> or even more. The mirror feature of lftp is extremely useful for such >> things. >> >> I wouldn't ever want having to mess around with web pages to figure out >> how to do this. Ftp is plain and simple. So you see why I'm explicitly >> asking for a replacement which is at least as good as ftp. >> >> >> -- >> "Didn't work" is an error. >> >> > Half petabyte datasets aren't really something I'd personally *ever* trust > ftp with in the first place. Why not? (12GB are nowhere close to half a petabyte ...) > That said, it depends entirely on the network > you're working with. Are you pushing this data in/out of the network your > machines live in, or are you working primarily internally? If internal, > what're the network side capabilities you have? Since you're likely > already > using something on the order of CEPH or Gluster to back the datasets where > they sit, just working with it all across network from that storage would > be my first instinct. The data would come in from suppliers. There isn't really anything going on atm but fetching data once a month which can be like 100MB or 12GB or more. That's because ppl don't use ftp ... >>> >>> I have the opposite experience. >>> I have the devil's own time trying to convince people to NOT use ftp for >>> anything and everything under the sun that even remotely resembles >>> getting data from A to B... >> >> I guess you're lucky then. >> >>> (especially things that are best done over a >>> message bus) >> >> Why would anyone try to transfer data over a message bus? Doesn't that >> require extra wiring and specialized hardware? >> >>> I'm still not understanding why you are asking your questions. What you >>> describe looks like the ideal case for ftp: >> >> it is >> >> Still nobody uses it, and apparently ftp usage is generally declining, >> so I would expect there to be a better alternative. >> >>> >>> - supplier pushes a file or files somewhere >>> - you fetch those files later at a suitable time >>> >>> it looks like a classic producer/consumer scenario and ftp or any of >>> it's webby clones like dropbox really it still the best tool overall. >>> Plus it has the added benefit that no user needs extra software - all >>> OSes have ftp clients even if it's just a browser >> >> The users don't know about that. >> >> > > > OK, so here is what you have. > > You apparently must use ftp as pretty much nothing else works. I was merely asking if there is a better solution. > You als claim that your users are too stupid to use ftp, and can't even > type ftp:// into a browser. I never said that. > I'm sorry, but that's only marginally more believable than claiming > keyboards are too complicated for your users. Does it matter what you or I believe? Some users have difficulties using a keyboard and/or a mouse. I've seen that, so no, what you or I believe does not matter. > I doubt anyone here can help you - you do not have a technical problem, > you have a people problem. I recommend the very first action you take > now is to critically re-examine why you think of those users in the way > you do and fix/amend what is going on inside your head. Then your way > for
Re: [gentoo-user] replacement for ftp?
On 03/05/2017 22:04, lee wrote: > Alan McKinnon writes: > >> On 30/04/2017 03:11, lee wrote: >>> "Poison BL." writes: >>> On Sat, Apr 29, 2017 at 3:24 PM, lee wrote: > Mick writes: > >> On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote: >>> On 25/04/2017 16:29, lee wrote: Hi, since the usage of FTP seems to be declining, what is a replacement which is at least as good as FTP? I'm aware that there's webdav, but that's very awkward to use and missing features. >>> >>> Why not stick with ftp? >>> Or, put another way, why do you feel you need to use something else? >>> >>> There's always dropbox >> >> >> Invariably all web hosting ISPs offer ftp(s) for file upload/download. > If you >> pay a bit more you should be able to get ssh/scp/sftp too. Indeed, many > ISPs >> throw in scp/sftp access as part of their basic package. >> >> Webdav(s) offers the same basic upload/download functionality, so I am > not >> sure what you find awkward about it, although I'd rather use lftp > instead of >> cadaver any day. ;-) >> >> As Alan mentioned, with JavaScript'ed web pages these days there are many >> webapp'ed ISP offerings like Dropbox and friends. >> >> What is the use case you have in mind? > > transferring large amounts of data and automatization in processing at > least some of it, without involving a 3rd party > > "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB, > or even more. The mirror feature of lftp is extremely useful for such > things. > > I wouldn't ever want having to mess around with web pages to figure out > how to do this. Ftp is plain and simple. So you see why I'm explicitly > asking for a replacement which is at least as good as ftp. > > > -- > "Didn't work" is an error. > > Half petabyte datasets aren't really something I'd personally *ever* trust ftp with in the first place. >>> >>> Why not? (12GB are nowhere close to half a petabyte ...) >>> That said, it depends entirely on the network you're working with. Are you pushing this data in/out of the network your machines live in, or are you working primarily internally? If internal, what're the network side capabilities you have? Since you're likely already using something on the order of CEPH or Gluster to back the datasets where they sit, just working with it all across network from that storage would be my first instinct. >>> >>> The data would come in from suppliers. There isn't really anything >>> going on atm but fetching data once a month which can be like 100MB or >>> 12GB or more. That's because ppl don't use ftp ... >> >> I have the opposite experience. >> I have the devil's own time trying to convince people to NOT use ftp for >> anything and everything under the sun that even remotely resembles >> getting data from A to B... > > I guess you're lucky then. > >> (especially things that are best done over a >> message bus) > > Why would anyone try to transfer data over a message bus? Doesn't that > require extra wiring and specialized hardware? > >> I'm still not understanding why you are asking your questions. What you >> describe looks like the ideal case for ftp: > > it is > > Still nobody uses it, and apparently ftp usage is generally declining, > so I would expect there to be a better alternative. > >> >> - supplier pushes a file or files somewhere >> - you fetch those files later at a suitable time >> >> it looks like a classic producer/consumer scenario and ftp or any of >> it's webby clones like dropbox really it still the best tool overall. >> Plus it has the added benefit that no user needs extra software - all >> OSes have ftp clients even if it's just a browser > > The users don't know about that. > > OK, so here is what you have. You apparently must use ftp as pretty much nothing else works. You als claim that your users are too stupid to use ftp, and can't even type ftp:// into a browser. I'm sorry, but that's only marginally more believable than claiming keyboards are too complicated for your users. I doubt anyone here can help you - you do not have a technical problem, you have a people problem. I recommend the very first action you take now is to critically re-examine why you think of those users in the way you do and fix/amend what is going on inside your head. Then your way forward will be clear. -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] replacement for ftp?
Alan McKinnon writes: > On 30/04/2017 03:11, lee wrote: >> "Poison BL." writes: >> >>> On Sat, Apr 29, 2017 at 3:24 PM, lee wrote: >>> Mick writes: > On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote: >> On 25/04/2017 16:29, lee wrote: >>> Hi, >>> >>> since the usage of FTP seems to be declining, what is a replacement >>> which is at least as good as FTP? >>> >>> I'm aware that there's webdav, but that's very awkward to use and >>> missing features. >> >> Why not stick with ftp? >> Or, put another way, why do you feel you need to use something else? >> >> There's always dropbox > > > Invariably all web hosting ISPs offer ftp(s) for file upload/download. If you > pay a bit more you should be able to get ssh/scp/sftp too. Indeed, many ISPs > throw in scp/sftp access as part of their basic package. > > Webdav(s) offers the same basic upload/download functionality, so I am not > sure what you find awkward about it, although I'd rather use lftp instead of > cadaver any day. ;-) > > As Alan mentioned, with JavaScript'ed web pages these days there are many > webapp'ed ISP offerings like Dropbox and friends. > > What is the use case you have in mind? transferring large amounts of data and automatization in processing at least some of it, without involving a 3rd party "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB, or even more. The mirror feature of lftp is extremely useful for such things. I wouldn't ever want having to mess around with web pages to figure out how to do this. Ftp is plain and simple. So you see why I'm explicitly asking for a replacement which is at least as good as ftp. -- "Didn't work" is an error. >>> Half petabyte datasets aren't really something I'd personally *ever* trust >>> ftp with in the first place. >> >> Why not? (12GB are nowhere close to half a petabyte ...) >> >>> That said, it depends entirely on the network >>> you're working with. Are you pushing this data in/out of the network your >>> machines live in, or are you working primarily internally? If internal, >>> what're the network side capabilities you have? Since you're likely already >>> using something on the order of CEPH or Gluster to back the datasets where >>> they sit, just working with it all across network from that storage would >>> be my first instinct. >> >> The data would come in from suppliers. There isn't really anything >> going on atm but fetching data once a month which can be like 100MB or >> 12GB or more. That's because ppl don't use ftp ... > > I have the opposite experience. > I have the devil's own time trying to convince people to NOT use ftp for > anything and everything under the sun that even remotely resembles > getting data from A to B... I guess you're lucky then. > (especially things that are best done over a > message bus) Why would anyone try to transfer data over a message bus? Doesn't that require extra wiring and specialized hardware? > I'm still not understanding why you are asking your questions. What you > describe looks like the ideal case for ftp: it is Still nobody uses it, and apparently ftp usage is generally declining, so I would expect there to be a better alternative. > > - supplier pushes a file or files somewhere > - you fetch those files later at a suitable time > > it looks like a classic producer/consumer scenario and ftp or any of > it's webby clones like dropbox really it still the best tool overall. > Plus it has the added benefit that no user needs extra software - all > OSes have ftp clients even if it's just a browser The users don't know about that. -- "Didn't work" is an error.
Re: [gentoo-user] replacement for ftp?
"Poison BL." writes: > On Sat, Apr 29, 2017 at 9:11 PM, lee wrote: >> >> "Poison BL." writes: >> > Half petabyte datasets aren't really something I'd personally *ever* > trust >> > ftp with in the first place. >> >> Why not? (12GB are nowhere close to half a petabyte ...) > > Ah... I completely misread that "or over 50k files in 12GB" as 50k files > *at* 12GB each... which works out to 0.6 PB, incidentally. > >> The data would come in from suppliers. There isn't really anything >> going on atm but fetching data once a month which can be like 100MB or >> 12GB or more. That's because ppl don't use ftp ... > > Really, if you're pulling it in from third party suppliers, you tend to be > tied to what they offer as a method of pulling it from them (or them > pushing it out to you), unless you're in the unique position to dictate the > decision for them. They need to use ftp to deliver the data, we need to use ftp to get the data. I don't want that any other way. The problem is that the ones supposed to deliver data are incompetent and don't want to use ftp because it's too complicated. So what's the better solution? > [...] > >> > How often does it need moved in/out of your facility, and is there no > way >> > to break up the processing into smaller chunks than a 0.6PB mass of > files? >> > Distribute out the smaller pieces with rsync, scp, or the like, operate > on >> > them, and pull back in the results, rather than trying to shift around > the >> > entire set. There's a reason Amazon will send a physical truck to a > site to >> > import large datasets into glacier... ;) >> >> Amazon has trucks? Perhaps they do in other countries. Here, amazon is >> just another web shop. They might have some delivery vans, but I've >> never seen one, so I doubt it. And why would anyone give them their >> data? There's no telling what they would do with it. > > Amazon's also one of the best known cloud computing suppliers on the planet > (AWS = Amazon Web Services). They have everything from pure compute > offerings to cloud storage geared towards *large* data archival. The latter > offering is named "glacier", and they offer a service for the import of > data into it (usually the "first pass", incremental changes are generally > done over the wire) that consists of a shipping truck with a rather nifty > storage system in the back of it that they hook right into your network. > You fill it with data, and then they drive it back to one of their data > centers to load it into place. They might not have that here. And who would want to give their data out of hands? -- "Didn't work" is an error.
Re: [gentoo-user] replacement for ftp?
"Walter Dnes" writes: >> transferring large amounts of data and automatization in processing at >> least some of it, without involving a 3rd party >> >> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB, >> or even more. The mirror feature of lftp is extremely useful for such >> things. >> >> I wouldn't ever want having to mess around with web pages to figure out >> how to do this. Ftp is plain and simple. So you see why I'm explicitly >> asking for a replacement which is at least as good as ftp. > > How about "wget"? It can handle ftp and http, and it can be scripted. Explain to someone unable to use Filezilla how to do that ... I'd have to look up if wget can do ftp with TLS. > And it can discriminate on timestamps, i.e. only download a file if it > has been changed since the latest download at your site. > > Then there's always "sneakernet". To quote Andrew Tanenbaum from 1981 > >> Never underestimate the bandwidth of a station wagon full of tapes >> hurtling down the highway. Hm, I'll suggest that, thanks. Ppl might be more likely to think they should be able to burn DVDs and send them in the mail than they are to think they could use something much simpler, easier, faster and more secure, like ftp ... -- "Didn't work" is an error.
Re: [gentoo-user] replacement for ftp?
Nils Freydank writes: > On Sat, 30 Apr 2017 19:04:06 +0200 Andrew Savchenko wrote: >> [...] >> I fail to see why FTP needs to be replaced: it works, it is >> supported, it is secure when used with care, it is damn fast. > > I’ll just drop the somewhat popular rant “FTP must die“[1] and a follow-up > discussion about it[2]. IMHO the main reasons are missing data integrity and > authentication security issues. The latter one can be solved with FTPS[3] - > but honestly > I never saw FTPS somewhere actually used in the wild. I've done it that way because I didn't want unencrypted transfers and sftp appears to be deprecated. It's working fine, and I don't want to replace it unless there were a better solution. > > > [1] http://mywiki.wooledge.org/FtpMustDie > [2] https://news.ycombinator.com/item?id=11251907 > [3] i.e. FTP over SSL/TLS (not to mix up with SFTP, which comes from the SSH > family) > > Greetings, > Nils -- "Didn't work" is an error.
Re: [gentoo-user] replacement for ftp?
On Monday 01 May 2017 22:36:00 Nils Freydank wrote: > On Sat, 30 Apr 2017 19:04:06 +0200 Andrew Savchenko wrote: > > [...] > > I fail to see why FTP needs to be replaced: it works, it is > > supported, it is secure when used with care, it is damn fast. > > I’ll just drop the somewhat popular rant “FTP must die“[1] and a follow-up > discussion about it[2]. IMHO the main reasons are missing data integrity and > authentication security issues. The latter one can be solved with FTPS[3] - > but honestly I never saw FTPS somewhere actually used in the wild. I'm not sure what you mean "used in the wild". I use lftp to connect via ftps with a number of webservers for updates and backups on a daily basis. Some of the connections are scripted. > [1] http://mywiki.wooledge.org/FtpMustDie > [2] https://news.ycombinator.com/item?id=11251907 > [3] i.e. FTP over SSL/TLS (not to mix up with SFTP, which comes from the SSH > family) > > Greetings, > Nils -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] replacement for ftp?
On Sat, 30 Apr 2017 19:04:06 +0200 Andrew Savchenko wrote: > [...] > I fail to see why FTP needs to be replaced: it works, it is > supported, it is secure when used with care, it is damn fast. I’ll just drop the somewhat popular rant “FTP must die“[1] and a follow-up discussion about it[2]. IMHO the main reasons are missing data integrity and authentication security issues. The latter one can be solved with FTPS[3] - but honestly I never saw FTPS somewhere actually used in the wild. [1] http://mywiki.wooledge.org/FtpMustDie [2] https://news.ycombinator.com/item?id=11251907 [3] i.e. FTP over SSL/TLS (not to mix up with SFTP, which comes from the SSH family) Greetings, Nils -- GPG fingerprint: '00EF D31F 1B60 D5DB ADB8 31C1 C0EC E696 0E54 475B' Nils Freydank signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] replacement for ftp?
On Tue, 25 Apr 2017 15:29:18 +0100 lee wrote: > > Hi, > > since the usage of FTP seems to be declining, what is a replacement > which is at least as good as FTP? I fail to see why FTP needs to be replaced: it works, it is supported, it is secure when used with care, it is damn fast. Best regards, Andrew Savchenko pgpH4meGDETra.pgp Description: PGP signature
Re: [gentoo-user] replacement for ftp?
On Sat, Apr 29, 2017 at 9:11 PM, lee wrote: > > "Poison BL." writes: > > Half petabyte datasets aren't really something I'd personally *ever* trust > > ftp with in the first place. > > Why not? (12GB are nowhere close to half a petabyte ...) Ah... I completely misread that "or over 50k files in 12GB" as 50k files *at* 12GB each... which works out to 0.6 PB, incidentally. > The data would come in from suppliers. There isn't really anything > going on atm but fetching data once a month which can be like 100MB or > 12GB or more. That's because ppl don't use ftp ... Really, if you're pulling it in from third party suppliers, you tend to be tied to what they offer as a method of pulling it from them (or them pushing it out to you), unless you're in the unique position to dictate the decision for them. From there, assuming you can push your choice of product on them, it becomes a question of how often the same dataset will need updated from the same sources, how much it changes between updates, how secure it needs to be in transit, how much you need to be able to trust that the source is still legitimately who you think it is, and how much verification that there wasn't any corruption during the transfer. Generic FTP has been losing favor over time because it was built in a time that many of those questions weren't really at the top of the list for concerns. SFTP (or SCP) (as long as keys are handled properly) allows for pretty solid certainty that a) both ends of the connection are who they say they are, b) those two ends are the only ones reading the data in transit, and c) the data that was sent is the same that was received (simply as a side benefit of the encryption/decryption). Rsync over SSH gives the same set of benefits, reduces the bandwidth used for updating the dataset (when it's the same dataset, at least), and will also verify the data on both ends (as it exists on disk) matches. If you're particularly lucky, the data might even hit just the right mark that benefits from the in-line compression you can turn on with SSH, too, cutting down the actual amount of bandwidth you burn through for each transfer. If your suppliers all have *nix based systems available, those are also standard tools that they'll have on hand. If they're strictly Windows shops, SCP/SFTP are still readily available, though they aren't built into the OS... rsync gets a bit trickier. > > How often does it need moved in/out of your facility, and is there no way > > to break up the processing into smaller chunks than a 0.6PB mass of files? > > Distribute out the smaller pieces with rsync, scp, or the like, operate on > > them, and pull back in the results, rather than trying to shift around the > > entire set. There's a reason Amazon will send a physical truck to a site to > > import large datasets into glacier... ;) > > Amazon has trucks? Perhaps they do in other countries. Here, amazon is > just another web shop. They might have some delivery vans, but I've > never seen one, so I doubt it. And why would anyone give them their > data? There's no telling what they would do with it. Amazon's also one of the best known cloud computing suppliers on the planet (AWS = Amazon Web Services). They have everything from pure compute offerings to cloud storage geared towards *large* data archival. The latter offering is named "glacier", and they offer a service for the import of data into it (usually the "first pass", incremental changes are generally done over the wire) that consists of a shipping truck with a rather nifty storage system in the back of it that they hook right into your network. You fill it with data, and then they drive it back to one of their data centers to load it into place. -- Poison [BLX] Joshua M. Murphy
Re: [gentoo-user] replacement for ftp?
On 30/04/2017 03:11, lee wrote: > "Poison BL." writes: > >> On Sat, Apr 29, 2017 at 3:24 PM, lee wrote: >> >>> Mick writes: >>> On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote: > On 25/04/2017 16:29, lee wrote: >> Hi, >> >> since the usage of FTP seems to be declining, what is a replacement >> which is at least as good as FTP? >> >> I'm aware that there's webdav, but that's very awkward to use and >> missing features. > > Why not stick with ftp? > Or, put another way, why do you feel you need to use something else? > > There's always dropbox Invariably all web hosting ISPs offer ftp(s) for file upload/download. >>> If you pay a bit more you should be able to get ssh/scp/sftp too. Indeed, many >>> ISPs throw in scp/sftp access as part of their basic package. Webdav(s) offers the same basic upload/download functionality, so I am >>> not sure what you find awkward about it, although I'd rather use lftp >>> instead of cadaver any day. ;-) As Alan mentioned, with JavaScript'ed web pages these days there are many webapp'ed ISP offerings like Dropbox and friends. What is the use case you have in mind? >>> >>> transferring large amounts of data and automatization in processing at >>> least some of it, without involving a 3rd party >>> >>> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB, >>> or even more. The mirror feature of lftp is extremely useful for such >>> things. >>> >>> I wouldn't ever want having to mess around with web pages to figure out >>> how to do this. Ftp is plain and simple. So you see why I'm explicitly >>> asking for a replacement which is at least as good as ftp. >>> >>> >>> -- >>> "Didn't work" is an error. >>> >>> >> Half petabyte datasets aren't really something I'd personally *ever* trust >> ftp with in the first place. > > Why not? (12GB are nowhere close to half a petabyte ...) > >> That said, it depends entirely on the network >> you're working with. Are you pushing this data in/out of the network your >> machines live in, or are you working primarily internally? If internal, >> what're the network side capabilities you have? Since you're likely already >> using something on the order of CEPH or Gluster to back the datasets where >> they sit, just working with it all across network from that storage would >> be my first instinct. > > The data would come in from suppliers. There isn't really anything > going on atm but fetching data once a month which can be like 100MB or > 12GB or more. That's because ppl don't use ftp ... I have the opposite experience. I have the devil's own time trying to convince people to NOT use ftp for anything and everything under the sun that even remotely resembles getting data from A to B... (especially things that are best done over a message bus) I'm still not understanding why you are asking your questions. What you describe looks like the ideal case for ftp: - supplier pushes a file or files somewhere - you fetch those files later at a suitable time it looks like a classic producer/consumer scenario and ftp or any of it's webby clones like dropbox really it still the best tool overall. Plus it has the added benefit that no user needs extra software - all OSes have ftp clients even if it's just a browser -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] replacement for ftp?
> transferring large amounts of data and automatization in processing at > least some of it, without involving a 3rd party > > "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB, > or even more. The mirror feature of lftp is extremely useful for such > things. > > I wouldn't ever want having to mess around with web pages to figure out > how to do this. Ftp is plain and simple. So you see why I'm explicitly > asking for a replacement which is at least as good as ftp. How about "wget"? It can handle ftp and http, and it can be scripted. And it can discriminate on timestamps, i.e. only download a file if it has been changed since the latest download at your site. Then there's always "sneakernet". To quote Andrew Tanenbaum from 1981 > Never underestimate the bandwidth of a station wagon full of tapes > hurtling down the highway. -- Walter Dnes I don't run "desktop environments"; I run useful applications
Re: [gentoo-user] replacement for ftp?
"Poison BL." writes: > On Sat, Apr 29, 2017 at 3:24 PM, lee wrote: > >> Mick writes: >> >> > On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote: >> >> On 25/04/2017 16:29, lee wrote: >> >> > Hi, >> >> > >> >> > since the usage of FTP seems to be declining, what is a replacement >> >> > which is at least as good as FTP? >> >> > >> >> > I'm aware that there's webdav, but that's very awkward to use and >> >> > missing features. >> >> >> >> Why not stick with ftp? >> >> Or, put another way, why do you feel you need to use something else? >> >> >> >> There's always dropbox >> > >> > >> > Invariably all web hosting ISPs offer ftp(s) for file upload/download. >> If you >> > pay a bit more you should be able to get ssh/scp/sftp too. Indeed, many >> ISPs >> > throw in scp/sftp access as part of their basic package. >> > >> > Webdav(s) offers the same basic upload/download functionality, so I am >> not >> > sure what you find awkward about it, although I'd rather use lftp >> instead of >> > cadaver any day. ;-) >> > >> > As Alan mentioned, with JavaScript'ed web pages these days there are many >> > webapp'ed ISP offerings like Dropbox and friends. >> > >> > What is the use case you have in mind? >> >> transferring large amounts of data and automatization in processing at >> least some of it, without involving a 3rd party >> >> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB, >> or even more. The mirror feature of lftp is extremely useful for such >> things. >> >> I wouldn't ever want having to mess around with web pages to figure out >> how to do this. Ftp is plain and simple. So you see why I'm explicitly >> asking for a replacement which is at least as good as ftp. >> >> >> -- >> "Didn't work" is an error. >> >> > Half petabyte datasets aren't really something I'd personally *ever* trust > ftp with in the first place. Why not? (12GB are nowhere close to half a petabyte ...) > That said, it depends entirely on the network > you're working with. Are you pushing this data in/out of the network your > machines live in, or are you working primarily internally? If internal, > what're the network side capabilities you have? Since you're likely already > using something on the order of CEPH or Gluster to back the datasets where > they sit, just working with it all across network from that storage would > be my first instinct. The data would come in from suppliers. There isn't really anything going on atm but fetching data once a month which can be like 100MB or 12GB or more. That's because ppl don't use ftp ... > How often does it need moved in/out of your facility, and is there no way > to break up the processing into smaller chunks than a 0.6PB mass of files? > Distribute out the smaller pieces with rsync, scp, or the like, operate on > them, and pull back in the results, rather than trying to shift around the > entire set. There's a reason Amazon will send a physical truck to a site to > import large datasets into glacier... ;) Amazon has trucks? Perhaps they do in other countries. Here, amazon is just another web shop. They might have some delivery vans, but I've never seen one, so I doubt it. And why would anyone give them their data? There's no telling what they would do with it. -- "Didn't work" is an error.
Re: [gentoo-user] replacement for ftp?
On Sat, Apr 29, 2017 at 3:24 PM, lee wrote: > Mick writes: > > > On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote: > >> On 25/04/2017 16:29, lee wrote: > >> > Hi, > >> > > >> > since the usage of FTP seems to be declining, what is a replacement > >> > which is at least as good as FTP? > >> > > >> > I'm aware that there's webdav, but that's very awkward to use and > >> > missing features. > >> > >> Why not stick with ftp? > >> Or, put another way, why do you feel you need to use something else? > >> > >> There's always dropbox > > > > > > Invariably all web hosting ISPs offer ftp(s) for file upload/download. > If you > > pay a bit more you should be able to get ssh/scp/sftp too. Indeed, many > ISPs > > throw in scp/sftp access as part of their basic package. > > > > Webdav(s) offers the same basic upload/download functionality, so I am > not > > sure what you find awkward about it, although I'd rather use lftp > instead of > > cadaver any day. ;-) > > > > As Alan mentioned, with JavaScript'ed web pages these days there are many > > webapp'ed ISP offerings like Dropbox and friends. > > > > What is the use case you have in mind? > > transferring large amounts of data and automatization in processing at > least some of it, without involving a 3rd party > > "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB, > or even more. The mirror feature of lftp is extremely useful for such > things. > > I wouldn't ever want having to mess around with web pages to figure out > how to do this. Ftp is plain and simple. So you see why I'm explicitly > asking for a replacement which is at least as good as ftp. > > > -- > "Didn't work" is an error. > > Half petabyte datasets aren't really something I'd personally *ever* trust ftp with in the first place. That said, it depends entirely on the network you're working with. Are you pushing this data in/out of the network your machines live in, or are you working primarily internally? If internal, what're the network side capabilities you have? Since you're likely already using something on the order of CEPH or Gluster to back the datasets where they sit, just working with it all across network from that storage would be my first instinct. How often does it need moved in/out of your facility, and is there no way to break up the processing into smaller chunks than a 0.6PB mass of files? Distribute out the smaller pieces with rsync, scp, or the like, operate on them, and pull back in the results, rather than trying to shift around the entire set. There's a reason Amazon will send a physical truck to a site to import large datasets into glacier... ;) -- Poison [BLX] Joshua M. Murphy
Re: [gentoo-user] replacement for ftp?
"Poison BL." writes: > On Tue, Apr 25, 2017 at 10:29 AM, lee wrote: > >> >> Hi, >> >> since the usage of FTP seems to be declining, what is a replacement >> which is at least as good as FTP? >> >> I'm aware that there's webdav, but that's very awkward to use and >> missing features. >> >> >> -- >> "Didn't work" is an error. >> >> > The one issue I have with all the answers I've seen is that they all lack > the most important question. You're asking for alternatives for an old tool > that was used for many use cases that, these days, have evolved to have > very different requirements for security, integration of access methods, > and general workflows for use. FTP used to be the go-to for long distance > file sharing for *all* use cases, one to one (user managing a website's > content), many to one (upload site), one to many (download site), etc. > What's your use case? all of them, with encrypted transfers and users needing a password for access I don't know anything better than ftp for this. Alternatively, there would need to be several different services for each group of users accommodating their particular use case, and being a nightmare to deploy, to maintain and to use. -- "Didn't work" is an error.
Re: [gentoo-user] replacement for ftp?
Alan McKinnon writes: > On 25/04/2017 16:29, lee wrote: >> >> Hi, >> >> since the usage of FTP seems to be declining, what is a replacement >> which is at least as good as FTP? >> >> I'm aware that there's webdav, but that's very awkward to use and >> missing features. >> >> > > Why not stick with ftp? The intended users are incompetent, hence it is too difficult to use ... > Or, put another way, why do you feel you need to use something else? I don't want to use anything else. Yet even Debian has announced that they will shut down their ftp services in November, one of the reasons being that almost no one uses them. Of course, their application is different from what I'm looking for because they only have downloads and no uploads. However, another reason given was that ftp isn't exactly friendly to firewalls and requires "awkward kludges" when load balancing is used. That is a pretty good reason. Anyway, when pretty much nobody uses a particular software anymore, it won't be very feasible to use that software. > There's always dropbox Well, dropbox sucks. I got a dropbox link and it didn't work at all, and handing out the data to some 3rd party is a very bad idea. It's also difficult to automate things with that. -- "Didn't work" is an error.
Re: [gentoo-user] replacement for ftp?
Danny YUE writes: > On 2017-04-25 14:29, lee wrote: >> Hi, >> >> since the usage of FTP seems to be declining, what is a replacement >> which is at least as good as FTP? >> >> I'm aware that there's webdav, but that's very awkward to use and >> missing features. > > What about sshfs? It allows you to mount a location that can be accessed > via ssh to your local file system, as if you are using ssh. Doesn't that require ssh access? And how do you explain that to ppl finding it too difficult to use Filezilla? Is it available for Windoze? > Also samba can be a replacement. I have a samba server on my OpenWRT > router and use mount.cifs to mount it... Does that work well, reliably and securely over internet connections? -- "Didn't work" is an error.
Re: [gentoo-user] replacement for ftp?
Mick writes: > On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote: >> On 25/04/2017 16:29, lee wrote: >> > Hi, >> > >> > since the usage of FTP seems to be declining, what is a replacement >> > which is at least as good as FTP? >> > >> > I'm aware that there's webdav, but that's very awkward to use and >> > missing features. >> >> Why not stick with ftp? >> Or, put another way, why do you feel you need to use something else? >> >> There's always dropbox > > > Invariably all web hosting ISPs offer ftp(s) for file upload/download. If > you > pay a bit more you should be able to get ssh/scp/sftp too. Indeed, many ISPs > throw in scp/sftp access as part of their basic package. > > Webdav(s) offers the same basic upload/download functionality, so I am not > sure what you find awkward about it, although I'd rather use lftp instead of > cadaver any day. ;-) > > As Alan mentioned, with JavaScript'ed web pages these days there are many > webapp'ed ISP offerings like Dropbox and friends. > > What is the use case you have in mind? transferring large amounts of data and automatization in processing at least some of it, without involving a 3rd party "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB, or even more. The mirror feature of lftp is extremely useful for such things. I wouldn't ever want having to mess around with web pages to figure out how to do this. Ftp is plain and simple. So you see why I'm explicitly asking for a replacement which is at least as good as ftp. -- "Didn't work" is an error.
Re: [gentoo-user] replacement for ftp?
On 2017-04-26 20:25, R0b0t1 wrote: > On Tue, Apr 25, 2017 at 8:26 PM, Danny YUE wrote: >> >> On 2017-04-25 19:59, R0b0t1 wrote: >>> On Tue, Apr 25, 2017 at 10:05 AM, Danny YUE wrote: On 2017-04-25 14:29, lee wrote: > Hi, > > since the usage of FTP seems to be declining, what is a replacement > which is at least as good as FTP? > > I'm aware that there's webdav, but that's very awkward to use and > missing features. What about sshfs? It allows you to mount a location that can be accessed via ssh to your local file system, as if you are using ssh. >>> >>> In a similar vein, scp. >> >> And considering something still robust but a little more smart, rsync. >> > > I was actually going to come back and suggest rscync over ssh. I > didn't originally mention it because I typically associate rsync with > backups. I recently found that rsnapshot (based on rsync) is a good and solid tool for backup...You may try it out ;-)
Re: [gentoo-user] replacement for ftp?
On Tue, Apr 25, 2017 at 10:29 AM, lee wrote: > > Hi, > > since the usage of FTP seems to be declining, what is a replacement > which is at least as good as FTP? > > I'm aware that there's webdav, but that's very awkward to use and > missing features. > > > -- > "Didn't work" is an error. > > The one issue I have with all the answers I've seen is that they all lack the most important question. You're asking for alternatives for an old tool that was used for many use cases that, these days, have evolved to have very different requirements for security, integration of access methods, and general workflows for use. FTP used to be the go-to for long distance file sharing for *all* use cases, one to one (user managing a website's content), many to one (upload site), one to many (download site), etc. What's your use case? -- Poison [BLX] Joshua M. Murphy
Re: [gentoo-user] replacement for ftp?
On Tue, Apr 25, 2017 at 8:26 PM, Danny YUE wrote: > > On 2017-04-25 19:59, R0b0t1 wrote: >> On Tue, Apr 25, 2017 at 10:05 AM, Danny YUE wrote: >>> >>> On 2017-04-25 14:29, lee wrote: Hi, since the usage of FTP seems to be declining, what is a replacement which is at least as good as FTP? I'm aware that there's webdav, but that's very awkward to use and missing features. >>> >>> What about sshfs? It allows you to mount a location that can be accessed >>> via ssh to your local file system, as if you are using ssh. >>> >> >> In a similar vein, scp. > > And considering something still robust but a little more smart, rsync. > I was actually going to come back and suggest rscync over ssh. I didn't originally mention it because I typically associate rsync with backups.
Re: [gentoo-user] replacement for ftp?
On 2017-04-25 19:59, R0b0t1 wrote: > On Tue, Apr 25, 2017 at 10:05 AM, Danny YUE wrote: >> >> On 2017-04-25 14:29, lee wrote: >>> Hi, >>> >>> since the usage of FTP seems to be declining, what is a replacement >>> which is at least as good as FTP? >>> >>> I'm aware that there's webdav, but that's very awkward to use and >>> missing features. >> >> What about sshfs? It allows you to mount a location that can be accessed >> via ssh to your local file system, as if you are using ssh. >> > > In a similar vein, scp. And considering something still robust but a little more smart, rsync.
Re: [gentoo-user] replacement for ftp?
On Tue, Apr 25, 2017 at 10:05 AM, Danny YUE wrote: > > On 2017-04-25 14:29, lee wrote: >> Hi, >> >> since the usage of FTP seems to be declining, what is a replacement >> which is at least as good as FTP? >> >> I'm aware that there's webdav, but that's very awkward to use and >> missing features. > > What about sshfs? It allows you to mount a location that can be accessed > via ssh to your local file system, as if you are using ssh. > In a similar vein, scp.
Re: [gentoo-user] replacement for ftp?
On 2017-04-25 14:29, lee wrote: > Hi, > > since the usage of FTP seems to be declining, what is a replacement > which is at least as good as FTP? > > I'm aware that there's webdav, but that's very awkward to use and > missing features. What about sshfs? It allows you to mount a location that can be accessed via ssh to your local file system, as if you are using ssh. Also samba can be a replacement. I have a samba server on my OpenWRT router and use mount.cifs to mount it... May these be helpful. Danny
Re: [gentoo-user] replacement for ftp?
On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote: > On 25/04/2017 16:29, lee wrote: > > Hi, > > > > since the usage of FTP seems to be declining, what is a replacement > > which is at least as good as FTP? > > > > I'm aware that there's webdav, but that's very awkward to use and > > missing features. > > Why not stick with ftp? > Or, put another way, why do you feel you need to use something else? > > There's always dropbox Invariably all web hosting ISPs offer ftp(s) for file upload/download. If you pay a bit more you should be able to get ssh/scp/sftp too. Indeed, many ISPs throw in scp/sftp access as part of their basic package. Webdav(s) offers the same basic upload/download functionality, so I am not sure what you find awkward about it, although I'd rather use lftp instead of cadaver any day. ;-) As Alan mentioned, with JavaScript'ed web pages these days there are many webapp'ed ISP offerings like Dropbox and friends. What is the use case you have in mind? -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] replacement for ftp?
On 25/04/2017 16:29, lee wrote: > > Hi, > > since the usage of FTP seems to be declining, what is a replacement > which is at least as good as FTP? > > I'm aware that there's webdav, but that's very awkward to use and > missing features. > > Why not stick with ftp? Or, put another way, why do you feel you need to use something else? There's always dropbox -- Alan McKinnon alan.mckin...@gmail.com