Re: [net] Problem with completePendingCommand and large files > 600MB
Note: curl supports an FTP range parameter.
It seems it does this by sending ABOR after the required data has arrived.
However I just tried and it does not work with the server I used.
You could try using it with your server and see if it works:
$ curl -v -r nnn-mmm -o file URL
If so you may be able to use ABOR in your app for that server.
On 25 October 2017 at 23:57, sebb wrote:
> What you are doing is stopping reading from the server before the file
> has been fully sent.
> Unless the server is expecting that, it may complain (the fact that it
> works for smaller files may be because the server has already sent the
> data).
>
> AFAICT there is no standard way to tell the server to only send a
> portion of the file.
>
> A work-round would be to fetch all the data but stop writing it to
> disk when you have got what you want.
>
> Another way might be to cancel the transfer after enough data has been
> received.
> However that relies on the server being able to process control
> channel requests whilst sending data.
>
> Or just live with the error message.
>
> It's not a bug in NET.
>
>
> On 25 October 2017 at 23:30, Bernd Eckenfels wrote:
>> 426 sounds like a server error, did you try to use a different client with
>> your server?
>>
>> Gruss
>> Bernd
>> --
>> http://bernd.eckenfels.net
>> ________
>> From: SeungWook Kim
>> Sent: Wednesday, October 25, 2017 6:34:02 PM
>> To: user@commons.apache.org
>> Subject: [net] Problem with completePendingCommand and large files > 600MB
>>
>> Hi,
>> I am having a problem with commons-net version* 3.6* and with large files
>>> 600MB and offset and length that does not include the whole file.
>>
>> I have included the test code below that appears to fail with large files >
>> 600 MB and an offset and length that does not include the whole file. If I
>> choose a smaller file like 90KB with offset and length, it works
>> successfully. If I choose a large file > 600MB with offset and length that
>> INCLUDES the entire file, it also works successfully. The
>> completePendingCommand replies back with "426 Failure writing network
>> stream" when the file is large > 600MB AND the offset and length is not the
>> entire file.
>>
>> I believe this is a bug but not sure. I did a quick look in the postings
>> and did not see anything that is exactly like it. Any feedback is greatly
>> appreciated.
>>
>>
>>
>> *** Need to specify serverName, userName,...absOutputFile below***
>> ***Start of Test code***
>>
>> public class FtpTest {
>> private static Logger log = LoggerFactory.getLogger(FtpTest.class);
>>
>> private static final int DEFAULT_TIMEOUT = 10*1000;
>>
>> @Before
>> public void setup() {
>>
>> }
>>
>> @Test
>> public void test() {
>> String serverName = "***";
>> String userName = "***";
>> String password = "***";
>> String absRemoteFileName = "***";
>> final long offset = ***;
>> final long length = ***;
>> String absOutputFile = "***";
>>
>> FTPClient ftpClient = new FTPClient();
>> try {
>> ftpClient.connect(serverName);
>> int reply = ftpClient.getReplyCode();
>> if (!FTPReply.isPositiveCompletion(reply)) {
>> String lastReply = ftpClient.getReplyString();
>> fail(lastReply);
>> }
>> ftpClient.addProtocolCommandListener(new
>> PrintCommandListener(new PrintWriter(System.out),
>> true));
>> ftpClient.setKeepAlive(true);
>> ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
>> //turn on SO_LINGER w/ timeout of 0 to cause connection to be
>> aborted immediately and to discard any
>> // pending data.
>> ftpClient.setSoLinger(true, 0);
>> if(ftpClient.login(userName, password)) {
>> ftpClient.enterLocalPassiveMode();
>> if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
>> {
>> log.info("Successfully connected in Passive Mode and
>> logged into {}", serverName);
>>
>> ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
>> ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
>> ftpClient.s
Re: [net] Problem with completePendingCommand and large files > 600MB
What you are doing is stopping reading from the server before the file
has been fully sent.
Unless the server is expecting that, it may complain (the fact that it
works for smaller files may be because the server has already sent the
data).
AFAICT there is no standard way to tell the server to only send a
portion of the file.
A work-round would be to fetch all the data but stop writing it to
disk when you have got what you want.
Another way might be to cancel the transfer after enough data has been received.
However that relies on the server being able to process control
channel requests whilst sending data.
Or just live with the error message.
It's not a bug in NET.
On 25 October 2017 at 23:30, Bernd Eckenfels wrote:
> 426 sounds like a server error, did you try to use a different client with
> your server?
>
> Gruss
> Bernd
> --
> http://bernd.eckenfels.net
>
> From: SeungWook Kim
> Sent: Wednesday, October 25, 2017 6:34:02 PM
> To: user@commons.apache.org
> Subject: [net] Problem with completePendingCommand and large files > 600MB
>
> Hi,
> I am having a problem with commons-net version* 3.6* and with large files
>> 600MB and offset and length that does not include the whole file.
>
> I have included the test code below that appears to fail with large files >
> 600 MB and an offset and length that does not include the whole file. If I
> choose a smaller file like 90KB with offset and length, it works
> successfully. If I choose a large file > 600MB with offset and length that
> INCLUDES the entire file, it also works successfully. The
> completePendingCommand replies back with "426 Failure writing network
> stream" when the file is large > 600MB AND the offset and length is not the
> entire file.
>
> I believe this is a bug but not sure. I did a quick look in the postings
> and did not see anything that is exactly like it. Any feedback is greatly
> appreciated.
>
>
>
> *** Need to specify serverName, userName,...absOutputFile below***
> ***Start of Test code***
>
> public class FtpTest {
> private static Logger log = LoggerFactory.getLogger(FtpTest.class);
>
> private static final int DEFAULT_TIMEOUT = 10*1000;
>
> @Before
> public void setup() {
>
> }
>
> @Test
> public void test() {
> String serverName = "***";
> String userName = "***";
> String password = "***";
> String absRemoteFileName = "***";
> final long offset = ***;
> final long length = ***;
> String absOutputFile = "***";
>
> FTPClient ftpClient = new FTPClient();
> try {
> ftpClient.connect(serverName);
> int reply = ftpClient.getReplyCode();
> if (!FTPReply.isPositiveCompletion(reply)) {
> String lastReply = ftpClient.getReplyString();
> fail(lastReply);
> }
> ftpClient.addProtocolCommandListener(new
> PrintCommandListener(new PrintWriter(System.out),
> true));
> ftpClient.setKeepAlive(true);
> ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
> //turn on SO_LINGER w/ timeout of 0 to cause connection to be
> aborted immediately and to discard any
> // pending data.
> ftpClient.setSoLinger(true, 0);
> if(ftpClient.login(userName, password)) {
> ftpClient.enterLocalPassiveMode();
> if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
> {
> log.info("Successfully connected in Passive Mode and
> logged into {}", serverName);
>
> ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
> ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
> ftpClient.setRestartOffset(offset);
> File writeToFile = new File(absOutputFile);
>
> try (InputStream inputStream =
> ftpClient.retrieveFileStream(absRemoteFileName);
> OutputStream outputStream =
> FileUtils.openOutputStream(writeToFile)) {
> if(inputStream == null || outputStream == null) {
> fail("One of the input/output stream is null");
> }
> final int BUFFER_SIZE = 4096;
> byte[] buffer = new byte[BUFFER_SIZE];
> int bytesRead;
> int len = (int)length;
> int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
> : l
Re: [net] Problem with completePendingCommand and large files > 600MB
426 sounds like a server error, did you try to use a different client with your
server?
Gruss
Bernd
--
http://bernd.eckenfels.net
From: SeungWook Kim
Sent: Wednesday, October 25, 2017 6:34:02 PM
To: user@commons.apache.org
Subject: [net] Problem with completePendingCommand and large files > 600MB
Hi,
I am having a problem with commons-net version* 3.6* and with large files
> 600MB and offset and length that does not include the whole file.
I have included the test code below that appears to fail with large files >
600 MB and an offset and length that does not include the whole file. If I
choose a smaller file like 90KB with offset and length, it works
successfully. If I choose a large file > 600MB with offset and length that
INCLUDES the entire file, it also works successfully. The
completePendingCommand replies back with "426 Failure writing network
stream" when the file is large > 600MB AND the offset and length is not the
entire file.
I believe this is a bug but not sure. I did a quick look in the postings
and did not see anything that is exactly like it. Any feedback is greatly
appreciated.
*** Need to specify serverName, userName,...absOutputFile below***
***Start of Test code***
public class FtpTest {
private static Logger log = LoggerFactory.getLogger(FtpTest.class);
private static final int DEFAULT_TIMEOUT = 10*1000;
@Before
public void setup() {
}
@Test
public void test() {
String serverName = "***";
String userName = "***";
String password = "***";
String absRemoteFileName = "***";
final long offset = ***;
final long length = ***;
String absOutputFile = "***";
FTPClient ftpClient = new FTPClient();
try {
ftpClient.connect(serverName);
int reply = ftpClient.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
String lastReply = ftpClient.getReplyString();
fail(lastReply);
}
ftpClient.addProtocolCommandListener(new
PrintCommandListener(new PrintWriter(System.out),
true));
ftpClient.setKeepAlive(true);
ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
//turn on SO_LINGER w/ timeout of 0 to cause connection to be
aborted immediately and to discard any
// pending data.
ftpClient.setSoLinger(true, 0);
if(ftpClient.login(userName, password)) {
ftpClient.enterLocalPassiveMode();
if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
{
log.info("Successfully connected in Passive Mode and
logged into {}", serverName);
ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
ftpClient.setRestartOffset(offset);
File writeToFile = new File(absOutputFile);
try (InputStream inputStream =
ftpClient.retrieveFileStream(absRemoteFileName);
OutputStream outputStream =
FileUtils.openOutputStream(writeToFile)) {
if(inputStream == null || outputStream == null) {
fail("One of the input/output stream is null");
}
final int BUFFER_SIZE = 4096;
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
int len = (int)length;
int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
: len;
while ((bytesRead = inputStream.read(buffer, 0,
bufferSize)) != -1) {
len -= bytesRead;
bufferSize = len > BUFFER_SIZE ? BUFFER_SIZE :
len;
outputStream.write(buffer, 0, bytesRead);
if(len <= 0) {
break;
}
}
} finally {
if(ftpClient.completePendingCommand()) {
log.info("Successful complete pending command");
} else {
fail("Unsuccessful complete pending command");
}
}
} else {
fail("Problem entering passive mode. All modern ftp
server supports passive mode, but " +
serverName + " doesn't??");
}
} else {
fail("Bad credentials to server " + serverName);
}
} catch (Exception e) {
fail(e.toString() + "\n" + e.getStackTrace().toString());
}
}
}
***End of Test code***
Thank you
-Seung
[net] Problem with completePendingCommand and large files > 600MB
Hi,
I am having a problem with commons-net version* 3.6* and with large files
> 600MB and offset and length that does not include the whole file.
I have included the test code below that appears to fail with large files >
600 MB and an offset and length that does not include the whole file. If I
choose a smaller file like 90KB with offset and length, it works
successfully. If I choose a large file > 600MB with offset and length that
INCLUDES the entire file, it also works successfully. The
completePendingCommand replies back with "426 Failure writing network
stream" when the file is large > 600MB AND the offset and length is not the
entire file.
I believe this is a bug but not sure. I did a quick look in the postings
and did not see anything that is exactly like it. Any feedback is greatly
appreciated.
*** Need to specify serverName, userName,...absOutputFile below***
***Start of Test code***
public class FtpTest {
private static Logger log = LoggerFactory.getLogger(FtpTest.class);
private static final int DEFAULT_TIMEOUT = 10*1000;
@Before
public void setup() {
}
@Test
public void test() {
String serverName = "***";
String userName = "***";
String password = "***";
String absRemoteFileName = "***";
final long offset = ***;
final long length = ***;
String absOutputFile = "***";
FTPClient ftpClient = new FTPClient();
try {
ftpClient.connect(serverName);
int reply = ftpClient.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
String lastReply = ftpClient.getReplyString();
fail(lastReply);
}
ftpClient.addProtocolCommandListener(new
PrintCommandListener(new PrintWriter(System.out),
true));
ftpClient.setKeepAlive(true);
ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
//turn on SO_LINGER w/ timeout of 0 to cause connection to be
aborted immediately and to discard any
// pending data.
ftpClient.setSoLinger(true, 0);
if(ftpClient.login(userName, password)) {
ftpClient.enterLocalPassiveMode();
if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
{
log.info("Successfully connected in Passive Mode and
logged into {}", serverName);
ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
ftpClient.setRestartOffset(offset);
File writeToFile = new File(absOutputFile);
try (InputStream inputStream =
ftpClient.retrieveFileStream(absRemoteFileName);
OutputStream outputStream =
FileUtils.openOutputStream(writeToFile)) {
if(inputStream == null || outputStream == null) {
fail("One of the input/output stream is null");
}
final int BUFFER_SIZE = 4096;
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
int len = (int)length;
int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
: len;
while ((bytesRead = inputStream.read(buffer, 0,
bufferSize)) != -1) {
len -= bytesRead;
bufferSize = len > BUFFER_SIZE ? BUFFER_SIZE :
len;
outputStream.write(buffer, 0, bytesRead);
if(len <= 0) {
break;
}
}
} finally {
if(ftpClient.completePendingCommand()) {
log.info("Successful complete pending command");
} else {
fail("Unsuccessful complete pending command");
}
}
} else {
fail("Problem entering passive mode. All modern ftp
server supports passive mode, but " +
serverName + " doesn't??");
}
} else {
fail("Bad credentials to server " + serverName);
}
} catch (Exception e) {
fail(e.toString() + "\n" + e.getStackTrace().toString());
}
}
}
***End of Test code***
Thank you
-Seung
