Re: Re: [net] FTPClient doesn't download full images
@Robert Paasche
I see.
Does your FTP server use Linux or Windows?
Something must have been stuck in RAM because after a PC reboot the success
rate (still in binary mode) with your very first tip (">=0" instead of ">0") is
a lot higher. It's still not even close to acceptable though:
With a buffer size (byte[]) of 1000 about 1-2 out of 20 downloaded files are
missing 1-2 bytes.
With a buffer size of 100k about 5-10 in 20 downloaded files are missing 1-4
bytes (sometimes more). I get images with a black bar/image artifacts again and
text files are also affected (missing 1-5 bytes).
With a buffer size of 100 the success rate is about has high as with 1000 and
the download obviously takes a lot longer.
The files with 1 missing bytes will still open fine and it doesn't look like
there's a problem (won't find any problems with that until I try to display the
images in my actual app) but it's nothing I can just ignore.
Thanks for correcting the bit, I tested it with `buffer.toByteArray()` instead:
The files will open now but about 2/3 of them are missing bytes (txt and
images) and there are image artifacts.
After receiving Erwin's e-mail I tested ASCII mode again: It looks like the
success rate with only txt files is 100% and it properly converts the original
files' "\r\n" (created with Windows) to Android's/Unix' "\n".
Binary mode is set properly, at least according to the return value of
`setFileType()`, but after all this testing the only explanation I can come up
with for this problem is that the library simply doesn't like FTP servers with
Windows in binary mode.
The only thing left to test now is using FTP commands instead of `retrieve` and
streams.
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
Re: [net] FTPClient doesn't download full images
@Erwin
Hi!
You can find the full code in the very first e-mail (May). I'm setting the
file type to `FTP.BINARY_FILE_TYPE` before I open the stream but when I'm
already connected to the server (as stated in the javadoc). I also tested it
with ASCII once, just to be sure that nothing was mixed up, but it obviously
didn't work.
-Ursprüngliche Nachricht-
Von: Erwin Hogeweg
Gesendet: Donnerstag, 4. Juni 2020 17:25
An: Commons Users List
Betreff: Re: [net] FTPClient doesn't download full images
Hi,
I have no idea if this is related but I seem to remember from a life long,
long ago that you can set up and FTP connection in two modes ASCII and
binary. The default is (used to be) ASCII and when you feed binary data
through that connection then (some) non-ascii characters were translated.
Again, I may be talking complete nonsense here but it may be worth to check
it out.
Cheers,
Erwin
> On Jun 4, 2020, at 09:29, Julia Ruzicka wrote:
>
> @Robert Paasche
>
> The code is used in an AsyncTask and there's more error handling to it
(including proper error messages,...), I just simplified it. ;)
>
> Have you ever tested it with images too? Text files always seem to
download properly (no matter what size), it's just the images that are
messed up.
>
> Thanks for the code snippet, I just ran my app with it multiple times and
the files are all the right size, even the images, but no program is able to
actually open them, so not sure what's going on with that now:
>
> if(singleFile.length > 0) { //check if file exists
>String localPath = localFolder + File.separator + filename;
>FTPFile single = singleFile[0];
>InputStream inStream = ftpClient.retrieveFileStream(filename);
>ByteArrayOutputStream buffer = new ByteArrayOutputStream();
>
>int nRead;
>byte[] data = new byte[(int) (single.getSize())]; //length was "*2" but
why?
>
>while ((nRead = inStream.read(data, 0, data.length)) != -1) {
>buffer.write(data, 0, nRead);
>}
>
>buffer.flush();
>inStream.close();
>
>FileOutputStream fos = new FileOutputStream(localPath);
>fos.write(data);
>fos.flush();
>fos.close();
>
>if(ftpClient.completePendingCommand()) {
>Log.d(TAG,"DONE");
>} else {
>Log.d(TAG,"NOT DONE");
>}
> }
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
Re: [net] FTPClient doesn't download full images
Hi,
I have no idea if this is related but I seem to remember from a life long, long
ago that you can set up and FTP connection in two modes ASCII and binary. The
default is (used to be) ASCII and when you feed binary data through that
connection then (some) non-ascii characters were translated.
Again, I may be talking complete nonsense here but it may be worth to check it
out.
Cheers,
Erwin
> On Jun 4, 2020, at 09:29, Julia Ruzicka wrote:
>
> @Robert Paasche
>
> The code is used in an AsyncTask and there's more error handling to it
> (including proper error messages,...), I just simplified it. ;)
>
> Have you ever tested it with images too? Text files always seem to download
> properly (no matter what size), it's just the images that are messed up.
>
> Thanks for the code snippet, I just ran my app with it multiple times and the
> files are all the right size, even the images, but no program is able to
> actually open them, so not sure what's going on with that now:
>
> if(singleFile.length > 0) { //check if file exists
>String localPath = localFolder + File.separator + filename;
>FTPFile single = singleFile[0];
>InputStream inStream = ftpClient.retrieveFileStream(filename);
>ByteArrayOutputStream buffer = new ByteArrayOutputStream();
>
>int nRead;
>byte[] data = new byte[(int) (single.getSize())]; //length was "*2" but
> why?
>
>while ((nRead = inStream.read(data, 0, data.length)) != -1) {
>buffer.write(data, 0, nRead);
>}
>
>buffer.flush();
>inStream.close();
>
>FileOutputStream fos = new FileOutputStream(localPath);
>fos.write(data);
>fos.flush();
>fos.close();
>
>if(ftpClient.completePendingCommand()) {
>Log.d(TAG,"DONE");
>} else {
>Log.d(TAG,"NOT DONE");
>}
> }
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
Re: Re: [net] FTPClient doesn't download full images
No never used with images.
//length was "*2" but why?
That is for an buggy ftp-server, which mishandled multibyte encodings. On
UTF-16 Files he has reported the number of chars instead of bytes, so the
file size was just the half erverytime.
fos.write(data);
this should be something like:
fos.write(buffer.toByteArray());
Am Do., 4. Juni 2020 um 15:29 Uhr schrieb Julia Ruzicka <
julia.ruzi...@simutech.at>:
> @Robert Paasche
>
> The code is used in an AsyncTask and there's more error handling to it
> (including proper error messages,...), I just simplified it. ;)
>
> Have you ever tested it with images too? Text files always seem to
> download properly (no matter what size), it's just the images that are
> messed up.
>
> Thanks for the code snippet, I just ran my app with it multiple times and
> the files are all the right size, even the images, but no program is able
> to actually open them, so not sure what's going on with that now:
>
> if(singleFile.length > 0) { //check if file exists
> String localPath = localFolder + File.separator + filename;
> FTPFile single = singleFile[0];
> InputStream inStream = ftpClient.retrieveFileStream(filename);
> ByteArrayOutputStream buffer = new ByteArrayOutputStream();
>
> int nRead;
> byte[] data = new byte[(int) (single.getSize())]; //length was "*2"
> but why?
>
> while ((nRead = inStream.read(data, 0, data.length)) != -1) {
> buffer.write(data, 0, nRead);
> }
>
> buffer.flush();
> inStream.close();
>
> FileOutputStream fos = new FileOutputStream(localPath);
> fos.write(data);
> fos.flush();
> fos.close();
>
> if(ftpClient.completePendingCommand()) {
> Log.d(TAG,"DONE");
> } else {
> Log.d(TAG,"NOT DONE");
> }
> }
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>
Re: Re: [net] FTPClient doesn't download full images
@Robert Paasche
The code is used in an AsyncTask and there's more error handling to it
(including proper error messages,...), I just simplified it. ;)
Have you ever tested it with images too? Text files always seem to download
properly (no matter what size), it's just the images that are messed up.
Thanks for the code snippet, I just ran my app with it multiple times and the
files are all the right size, even the images, but no program is able to
actually open them, so not sure what's going on with that now:
if(singleFile.length > 0) { //check if file exists
String localPath = localFolder + File.separator + filename;
FTPFile single = singleFile[0];
InputStream inStream = ftpClient.retrieveFileStream(filename);
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[(int) (single.getSize())]; //length was "*2" but why?
while ((nRead = inStream.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
inStream.close();
FileOutputStream fos = new FileOutputStream(localPath);
fos.write(data);
fos.flush();
fos.close();
if(ftpClient.completePendingCommand()) {
Log.d(TAG,"DONE");
} else {
Log.d(TAG,"NOT DONE");
}
}
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
Re: Re: [net] FTPClient doesn't download full images
@ julia.ruzicka content is just a simple POJO holding the byte and meta data (like timestamps). In some situations I'm using a fileoutputstream after disconnecting from ftp. I'm downloading large (>300 mb) zip-Files. Am Do., 4. Juni 2020 um 14:10 Uhr schrieb Julia Ruzicka < julia.ruzi...@simutech.at>: > @Robert Paasche > > At first I tried to do it with a `FileOutputStream` and > `ftpClient.retrieveFile(fileName,fos)` but that's how I noticed the > problem. I then switched to using ` InputStream is = > ftpClient.retrieveFileStream(filename)` and when that didn't work properly > either, I added ` BufferedInputStream bis = new BufferedInputStream(is)`. > > What type of files are you downloading? > > What's your `content` and how do you write the data? With a > DataOutputStream? > > > > - > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org > For additional commands, e-mail: user-h...@commons.apache.org > >
Re: [net] FTPClient doesn't download full images
Don't know if the code you have showed is your original one but in general its not a good idea to "catch and ignore" exceptions, yo either: - simply buble up to the caller - wrap into other exceptions if required by the API - explicitly handle/document I'm using comons-ftp for large files and small files and it always has worked quite well. Am 04.06.20 um 14:10 schrieb Julia Ruzicka: @Robert Paasche At first I tried to do it with a `FileOutputStream` and `ftpClient.retrieveFile(fileName,fos)` but that's how I noticed the problem. I then switched to using ` InputStream is = ftpClient.retrieveFileStream(filename)` and when that didn't work properly either, I added ` BufferedInputStream bis = new BufferedInputStream(is)`. What type of files are you downloading? What's your `content` and how do you write the data? With a DataOutputStream? - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
Re: Re: [net] FTPClient doesn't download full images
@Robert Paasche At first I tried to do it with a `FileOutputStream` and `ftpClient.retrieveFile(fileName,fos)` but that's how I noticed the problem. I then switched to using ` InputStream is = ftpClient.retrieveFileStream(filename)` and when that didn't work properly either, I added ` BufferedInputStream bis = new BufferedInputStream(is)`. What type of files are you downloading? What's your `content` and how do you write the data? With a DataOutputStream? - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
Re: [net] FTPClient doesn't download full images
If you want to read a stream without dealing with low-level headaches, you
can use
https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/IOUtils.html
Gary
On Thu, Jun 4, 2020 at 6:53 AM Julia Ruzicka
wrote:
> Thanks for your reply!
>
> I changed the code and did the usual AndroidStudio restart/clean/build but
> most images are still downloaded incompletely.
> I don't know if my other answer (sent about an hour ago) came through (I
> did some more testing today), so here's an imgur album with test images and
> their results:
> https://imgur.com/a/Mzngp11
>
> The last image (the satellite one) was now also affected for the first
> time.
>
> -Ursprüngliche Nachricht-
> Von: Robert Paasche
> Gesendet: Donnerstag, 4. Juni 2020 12:18
> An: Commons Users List
> Betreff: Re: [net] FTPClient doesn't download full images
>
> The following line is wrong:
> while ((len = bis.read(buf)) > 0) { //Same result using only 'is'
>
> read == -1 marks the end of the stream. 0 shows only that the stream is
> still waiting for the next chunk of data.
>
> Please read the JavaDocs:
>
> > Returns:
> > the number of bytes read, or -1 if the end of the stream has been
> reached.
>
>
> while ((len = bis.read(buf)) >= 0) { //Same result using only 'is'
> if (len >0)
>out.write(buf, 0, len);
> else
> // prevent high CPU load
> LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
> }
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>
>
Re: [net] FTPClient doesn't download full images
Hmm maybe it is the bufferdStream.
That's working code I'm using in a project for years now:
private FTPClient createClient()
{
FTPClient ftp = new FTPClient();
FTPClientConfig config = new FTPClientConfig();
config.setServerTimeZoneId("UTC");
ftp.setControlKeepAliveTimeout(1000 * 60 * 5l);
ftp.setControlKeepAliveReplyTimeout(1000 * 60 * 5);
ftp.setDataTimeout(1000 * 60 * 5);
ftp.setRemoteVerificationEnabled(false);
return ftp;
}
if (FetchMode.BYTE_DATA.equals(fetchMode))
{
LOGGER.info("Using BINARY_FILE_TYPE.");
ftp.setFileType(FTP.BINARY_FILE_TYPE);
InputStream inStream = ftp.retrieveFileStream(ftpFile.getName());
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[(int) (ftpFile.getSize() * 2)];
while ((nRead = inStream.read(data, 0, data.length)) != -1)
{
buffer.write(data, 0, nRead);
}
buffer.flush();
content.setByteData(buffer.toByteArray());
}
Re: [net] FTPClient doesn't download full images
Thanks for your reply!
I changed the code and did the usual AndroidStudio restart/clean/build but most
images are still downloaded incompletely.
I don't know if my other answer (sent about an hour ago) came through (I did
some more testing today), so here's an imgur album with test images and their
results:
https://imgur.com/a/Mzngp11
The last image (the satellite one) was now also affected for the first time.
-Ursprüngliche Nachricht-
Von: Robert Paasche
Gesendet: Donnerstag, 4. Juni 2020 12:18
An: Commons Users List
Betreff: Re: [net] FTPClient doesn't download full images
The following line is wrong:
while ((len = bis.read(buf)) > 0) { //Same result using only 'is'
read == -1 marks the end of the stream. 0 shows only that the stream is still
waiting for the next chunk of data.
Please read the JavaDocs:
> Returns:
> the number of bytes read, or -1 if the end of the stream has been reached.
while ((len = bis.read(buf)) >= 0) { //Same result using only 'is'
if (len >0)
out.write(buf, 0, len);
else
// prevent high CPU load
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
}
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
Re: [net] FTPClient doesn't download full images
The following line is wrong:
while ((len = bis.read(buf)) > 0) { //Same result using only 'is'
read == -1 marks the end of the stream. 0 shows only that the stream is
still waiting for the next chunk of data.
Please read the JavaDocs:
> Returns:
> the number of bytes read, or -1 if the end of the stream has been reached.
while ((len = bis.read(buf)) >= 0) { //Same result using only 'is'
if (len >0)
out.write(buf, 0, len);
else
// prevent high CPU load
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
}
AW: [net] FTPClient doesn't download full images
Hi everyone! I created a small test app that uses my "download" class on a button press (but doesn't do anything else) and did some more testing with it (with "FTP.BINARY_FILE_TYPE"): - Downloading a 14mb txt file (and its copies) always finishes with a complete file (even with a long file name) - Downloading a 12kb jpg (and its copies) always finishes with a complete file - Downloading the 8mb (png) or 6mb (jpg version) images sometimes completes but most of the time it doesn't - the file size is now always the same but there's still a black bar at the bottom of the images (height differs) - I uploaded a couple of random google images to my FTP server and most of them are affected by this problem too. You can find all the tested google images (including the sources) and the results in this imgur album: https://imgur.com/a/Mzngp11 I uploaded a couple of the images to a public FTP test server (https://dlptest.com/ftp-test/) and I get the same results with it. Has anyone got an idea what's wrong and how do fix this? - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
[net] FTPClient doesn't download full images
Hi everyone!
This is my first time using a mailing list, so sorry in advance if I mess
something up!
I'm using version 3.6 of the "Net" Library, in combination with Java 8,
Android Studio 3.6.1 (min SDK=24, target SDK=27) and I'm experiencing a
weird issue with file download:
Downloading small images (jpg, around 12kb each) or text files from my FTP
server always succeeds but I've also got bigger images (png, 2000x2000,
around 8-9mb each) and they only download partially: I pulled them from the
emulator/device and they end up with a black, horizontal bar at the bottom
and they're smaller than they should be (see below for examples).
I tested my code with 'ftpClient.retrieveFile()' (version 1) and
'ftpClient.retrieveFileStream()' (version 2, InputStream or
BufferedInputStream) but I get the same result with both:
public void downloadFiles(String remoteFolder, String localFolder,
ArrayList filenames) { //This runs inside an 'AsyncTask'!
//login here
ftpClient.setConnectTimeout(1);
ftpClient.setDefaultTimeout(3);
OutputStream out = null;
try {
ftpClient.enterLocalPassiveMode();
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
if(ftpClient.changeWorkingDirectory(remoteFolder)) {
for(String filename : filenames) {
FTPFile[] singleFile = ftpClient.listFiles(filename);
if(singleFile.length > 0) { //check if file exists
String localPath = localFolder + File.separator +
filename;
out = new FileOutputStream(localPath);
//Version 1:
/*if(!ftpClient.retrieveFile(filename, out)) {
//Set error message here
out.close();
break;
}
out.close();*/
//Version 2:
InputStream is = ftpClient.retrieveFileStream(filename);
BufferedInputStream bis = new BufferedInputStream(is);
byte[] buf = new byte[1000];
int len;
while ((len = bis.read(buf)) > 0) { //Same result using
only 'is'
out.write(buf, 0, len);
}
if(ftpClient.completePendingCommand()) {
Log.d(TAG,"DONE"); //This is always printed, even
when the files aren't complete
} else {
Log.d(TAG,"NOT DONE");
}
out.close();
bis.close();
} else {
//Another error message here
break;
}
}
}
} catch (IOException e) {
//And another error message here
} finally {
try {
if(out!=null) { out.close(); }
} catch(IOException e) {
//Doesn't matter
}
}
//logout here
}
I printed the full length (added up 'len') and only very rarely it manages
to download everything, usually there are a couple of bytes missing (always
less than 10 so far) - example:
File 1 is 8965167 bytes (according to Windows' file explorer) but the added
up 'len' is only 8965165 or 8965161 or .
File 2 is 8972047 bytes but it only downloads 8972044 or 8972038 or .
Did I miss anything, some type of setting for bigger files? How do I make it
download the full images?
Thanks in advance for any tip!
