Re: NPE in getRemoteAdress

[sebb]

FTPClient instances are independent; each can be used by a separate thread.
However note that the instances are not inherently thread-safe.

So either create the instance in the same thread that does the download.

Alternatively, create the client before creating the thread, and
ensure that the client is only used by the new thread once created.

On 30 August 2016 at 20:53, Oliver Zemann <oliver.zem...@gmail.com> wrote:
> Seems like that made it. I  used a new FTPClient for each file to download,
> that worked now in the JUnit. Will try it this weekend to get the same in my
> application. Thanks!
>
>
> Am 30.08.2016 um 16:23 schrieb sebb:
>>
>> On 29 August 2016 at 18:28, Oliver Zemann <oliver.zem...@gmail.com> wrote:
>>>
>>> My download method looks like this:
>>>
>>> private void downloadDirectory(FTPClient client, String sourceDir, String
>>> destDir,
>>> ScheduledExecutorService
>>> scheduledExecutorService) throws IOException {
>>>
>>>  final FTPFile[] ftpFiles = client.listFiles(sourceDir);
>>>  for (final FTPFile ftpFile : ftpFiles) {
>>>
>>>  //skip . and ..
>>>  if (ftpFile.isDirectory() && (ftpFile.getName().equals(".")
>>> ||
>>> ftpFile.getName().equals(".."))) {
>>>  continue;
>>>  }
>>>
>>>  if (ftpFile.isDirectory()) {
>>>  downloadDirectory(client, sourceDir + ftpFile.getName(),
>>> destDir + "/" + ftpFile.getName(), scheduledExecutorService);
>>>  } else {
>>> //scheduledExecutorService.submit(new
>>> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>>>  FileOutputStream fos = new
>>> FileOutputStream(ftpFile.getName());
>>>  client.retrieveFile(sourceDir + ftpFile.getName(), fos
>>> );
>>>  fos.flush();
>>>  fos.close();
>>>  }
>>>  }
>>>  }
>>>
>>> When i run it like this, it works and it downloads the files. But when i
>>> use
>>> the scheduledExecuterService, it fails.
>>> So i replaced the 2 lines with my download method (where it uses the
>>> inputstream) to see what happens:
>>>
>>>   } else {
>>> //scheduledExecutorService.submit(new
>>> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>>>  new File(destDir).mkdirs();
>>>  final File outFile = new File(destDir,
>>> ftpFile.getName());
>>>  final FileOutputStream fos = new
>>> FileOutputStream(outFile);
>>>  final InputStream inputStream =
>>> client.retrieveFileStream(sourceDir + ftpFile.getName());
>>>
>>>  byte[] buffer = new byte[8096];
>>>  int len = -1;
>>>  try {
>>>  len = inputStream.read(buffer, 0, buffer.length);
>>>  } catch (Exception ex) {
>>>  ex.printStackTrace();
>>>  }
>>>  while (len != -1) {
>>>  fos.write(buffer, 0, len);
>>>  len = inputStream.read(buffer);
>>>  if (Thread.interrupted()) {
>>>  throw new InterruptedException();
>>>  }
>>>  }
>>>  client.completePendingCommand();
>>> //informListenersWeFinishedOneFile(sourceDir +
>>> ftpFile.getName(), bytesWritten);
>>>  fos.flush();
>>>  fos.close();
>>>  }
>>>  }
>>>
>>> That works too?!
>>>
>>> I will go deeper into that at weekend and check whats going on, but at
>>> the
>>> moment i really dont understand it. Maybe some kind of concurrency
>>> problem
>>> with FTPClient?
>>
>> That seems the most likely.
>>
>> FTPClient is not guaranteed to be thread-safe.
>> I'm not sure that the java.io classes such as InputStream are thread
>> safe either.
>>
>>> Am 29.08.2016 um 01:40 schrieb Martin Gainty:
>>>>
>>>> possible timeout waiting for FTP to reply
>>>> examples.ftp.FTPClientExample says to increase FTP reply tim

Re: NPE in getRemoteAdress

[Oliver Zemann]

Seems like that made it. I  used a new FTPClient for each file to 
download, that worked now in the JUnit. Will try it this weekend to get 
the same in my application. Thanks!



Am 30.08.2016 um 16:23 schrieb sebb:

On 29 August 2016 at 18:28, Oliver Zemann <oliver.zem...@gmail.com> wrote:

My download method looks like this:

private void downloadDirectory(FTPClient client, String sourceDir, String
destDir,
ScheduledExecutorService
scheduledExecutorService) throws IOException {

 final FTPFile[] ftpFiles = client.listFiles(sourceDir);
 for (final FTPFile ftpFile : ftpFiles) {

 //skip . and ..
 if (ftpFile.isDirectory() && (ftpFile.getName().equals(".") ||
ftpFile.getName().equals(".."))) {
 continue;
 }

 if (ftpFile.isDirectory()) {
 downloadDirectory(client, sourceDir + ftpFile.getName(),
destDir + "/" + ftpFile.getName(), scheduledExecutorService);
 } else {
//scheduledExecutorService.submit(new
DownloadRunnable(destDir, ftpFile, sourceDir, client));
 FileOutputStream fos = new
FileOutputStream(ftpFile.getName());
 client.retrieveFile(sourceDir + ftpFile.getName(), fos );
 fos.flush();
 fos.close();
 }
 }
 }

When i run it like this, it works and it downloads the files. But when i use
the scheduledExecuterService, it fails.
So i replaced the 2 lines with my download method (where it uses the
inputstream) to see what happens:

  } else {
//scheduledExecutorService.submit(new
DownloadRunnable(destDir, ftpFile, sourceDir, client));
 new File(destDir).mkdirs();
 final File outFile = new File(destDir, ftpFile.getName());
 final FileOutputStream fos = new FileOutputStream(outFile);
 final InputStream inputStream =
client.retrieveFileStream(sourceDir + ftpFile.getName());

 byte[] buffer = new byte[8096];
 int len = -1;
 try {
 len = inputStream.read(buffer, 0, buffer.length);
 } catch (Exception ex) {
 ex.printStackTrace();
 }
 while (len != -1) {
 fos.write(buffer, 0, len);
 len = inputStream.read(buffer);
 if (Thread.interrupted()) {
 throw new InterruptedException();
 }
 }
 client.completePendingCommand();
//informListenersWeFinishedOneFile(sourceDir +
ftpFile.getName(), bytesWritten);
 fos.flush();
 fos.close();
 }
 }

That works too?!

I will go deeper into that at weekend and check whats going on, but at the
moment i really dont understand it. Maybe some kind of concurrency problem
with FTPClient?

That seems the most likely.

FTPClient is not guaranteed to be thread-safe.
I'm not sure that the java.io classes such as InputStream are thread
safe either.


Am 29.08.2016 um 01:40 schrieb Martin Gainty:

possible timeout waiting for FTP to reply
examples.ftp.FTPClientExample says to increase FTP reply timeout with -w
parameter
   if (args[base].equals("-w")) {
controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]);}
or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec
reply timeout
?
Martin
__




Date: Sun, 28 Aug 2016 20:06:46 +0200
From: e...@zusammenkunft.net
To: user@commons.apache.org
CC: oliver.zem...@gmail.com
Subject: Re: NPE in getRemoteAdress

Hello,

I am not sure about your NPE, but this code here ignores the result of
the read call. It cannot deal with short reads:

   Am Sun, 28 Aug 2016
15:50:36 +0200 schrieb Oliver Zemann <oliver.zem...@gmail.com>:

byte b[] =new byte[4096];
while (inputStream.read(b) != -1) {
fos.write(b);
bytesWritten += b.length;


Gruss
Bernd

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: NPE in getRemoteAdress

[Oliver Zemann]

rieveFile(sourceDir + ftpFile.getName(), fos );
 fos.flush();
 fos.close();
 }
 }
 }

When i run it like this, it works and it downloads the files. But when i use
the scheduledExecuterService, it fails.
So i replaced the 2 lines with my download method (where it uses the
inputstream) to see what happens:

  } else {
//scheduledExecutorService.submit(new
DownloadRunnable(destDir, ftpFile, sourceDir, client));
 new File(destDir).mkdirs();
 final File outFile = new File(destDir, ftpFile.getName());
 final FileOutputStream fos = new FileOutputStream(outFile);
 final InputStream inputStream =
client.retrieveFileStream(sourceDir + ftpFile.getName());

 byte[] buffer = new byte[8096];
 int len = -1;
 try {
 len = inputStream.read(buffer, 0, buffer.length);
 } catch (Exception ex) {
 ex.printStackTrace();
 }
 while (len != -1) {
 fos.write(buffer, 0, len);
 len = inputStream.read(buffer);
 if (Thread.interrupted()) {
 throw new InterruptedException();
 }
 }
 client.completePendingCommand();
//informListenersWeFinishedOneFile(sourceDir +
ftpFile.getName(), bytesWritten);
 fos.flush();
 fos.close();
 }
 }

That works too?!

I will go deeper into that at weekend and check whats going on, but at the
moment i really dont understand it. Maybe some kind of concurrency problem
with FTPClient?

That seems the most likely.

FTPClient is not guaranteed to be thread-safe.
I'm not sure that the java.io classes such as InputStream are thread
safe either.


Am 29.08.2016 um 01:40 schrieb Martin Gainty:

possible timeout waiting for FTP to reply
examples.ftp.FTPClientExample says to increase FTP reply timeout with -w
parameter
   if (args[base].equals("-w")) {
controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]);}
or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec
reply timeout
?
Martin
__




Date: Sun, 28 Aug 2016 20:06:46 +0200
From: e...@zusammenkunft.net
To: user@commons.apache.org
CC: oliver.zem...@gmail.com
Subject: Re: NPE in getRemoteAdress

Hello,

I am not sure about your NPE, but this code here ignores the result of
the read call. It cannot deal with short reads:

   Am Sun, 28 Aug 2016
15:50:36 +0200 schrieb Oliver Zemann <oliver.zem...@gmail.com>:

byte b[] =new byte[4096];
while (inputStream.read(b) != -1) {
fos.write(b);
bytesWritten += b.length;


Gruss
Bernd

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org


-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: NPE in getRemoteAdress

[sebb]

On 29 August 2016 at 18:28, Oliver Zemann <oliver.zem...@gmail.com> wrote:
> My download method looks like this:
>
> private void downloadDirectory(FTPClient client, String sourceDir, String
> destDir,
>ScheduledExecutorService
> scheduledExecutorService) throws IOException {
>
> final FTPFile[] ftpFiles = client.listFiles(sourceDir);
> for (final FTPFile ftpFile : ftpFiles) {
>
> //skip . and ..
> if (ftpFile.isDirectory() && (ftpFile.getName().equals(".") ||
> ftpFile.getName().equals(".."))) {
> continue;
> }
>
> if (ftpFile.isDirectory()) {
> downloadDirectory(client, sourceDir + ftpFile.getName(),
> destDir + "/" + ftpFile.getName(), scheduledExecutorService);
> } else {
> //scheduledExecutorService.submit(new
> DownloadRunnable(destDir, ftpFile, sourceDir, client));
> FileOutputStream fos = new
> FileOutputStream(ftpFile.getName());
> client.retrieveFile(sourceDir + ftpFile.getName(), fos );
> fos.flush();
> fos.close();
> }
> }
> }
>
> When i run it like this, it works and it downloads the files. But when i use
> the scheduledExecuterService, it fails.
> So i replaced the 2 lines with my download method (where it uses the
> inputstream) to see what happens:
>
>  } else {
> //scheduledExecutorService.submit(new
> DownloadRunnable(destDir, ftpFile, sourceDir, client));
> new File(destDir).mkdirs();
> final File outFile = new File(destDir, ftpFile.getName());
> final FileOutputStream fos = new FileOutputStream(outFile);
> final InputStream inputStream =
> client.retrieveFileStream(sourceDir + ftpFile.getName());
>
> byte[] buffer = new byte[8096];
> int len = -1;
> try {
> len = inputStream.read(buffer, 0, buffer.length);
> } catch (Exception ex) {
> ex.printStackTrace();
> }
> while (len != -1) {
> fos.write(buffer, 0, len);
> len = inputStream.read(buffer);
> if (Thread.interrupted()) {
> throw new InterruptedException();
> }
> }
> client.completePendingCommand();
> //informListenersWeFinishedOneFile(sourceDir +
> ftpFile.getName(), bytesWritten);
> fos.flush();
> fos.close();
> }
> }
>
> That works too?!
>
> I will go deeper into that at weekend and check whats going on, but at the
> moment i really dont understand it. Maybe some kind of concurrency problem
> with FTPClient?

That seems the most likely.

FTPClient is not guaranteed to be thread-safe.
I'm not sure that the java.io classes such as InputStream are thread
safe either.

>
> Am 29.08.2016 um 01:40 schrieb Martin Gainty:
>>
>> possible timeout waiting for FTP to reply
>> examples.ftp.FTPClientExample says to increase FTP reply timeout with -w
>> parameter
>>   if (args[base].equals("-w")) {
>> controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]);}
>> or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec
>> reply timeout
>> ?
>> Martin
>> __
>>
>>
>>
>>> Date: Sun, 28 Aug 2016 20:06:46 +0200
>>> From: e...@zusammenkunft.net
>>> To: user@commons.apache.org
>>> CC: oliver.zem...@gmail.com
>>> Subject: Re: NPE in getRemoteAdress
>>>
>>> Hello,
>>>
>>> I am not sure about your NPE, but this code here ignores the result of
>>> the read call. It cannot deal with short reads:
>>>
>>>   Am Sun, 28 Aug 2016
>>> 15:50:36 +0200 schrieb Oliver Zemann <oliver.zem...@gmail.com>:
>>>>
>>>> byte b[] =new byte[4096];
>>>> while (inputStream.read(b) != -1) {
>>>>fos.write(b);
>>>>bytesWritten += b.length;
>>>
>>>
>>> Gruss
>>> Bernd
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
>>> For additional commands, e-mail: user-h...@commons.apache.org
>>>
>>
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
>

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

RE: NPE in getRemoteAdress

[Martin Gainty]

if socket can stay open for normal read then that should work for filters or 
scheduled tasks
can you put project on github or any accessible URL and send me a link or plain 
attach a zip and I'll take a look

Thanks
Martin 
__ 



> Subject: Re: NPE in getRemoteAdress
> To: user@commons.apache.org
> From: oliver.zem...@gmail.com
> Date: Mon, 29 Aug 2016 21:27:46 +0200
> 
> Hi Martin
> 
> The ScheduledExecutorService is used in my project and is part of the 
> JDK/JRE since version 7. This class is probably not in apache commons 
> net ftp, i dont know how you got that information? Guess a misunderstanding.
> 
> I read the documentation. I set the timeout of the ftp client to 20 
> seconds which should be long enough. Also, i see when it jumps inside my 
> threads and executes them, and here is the problem. The socket is 
> closed, which should _not_ be closed. But i dont know what is causing 
> this wierd behaviour.
> 
> The project is open source. If you would like to take a look at it, i 
> can send you the project url by private mail.
> 
> Regards
> 
> Oli
> 
> 
> Am 29.08.2016 um 21:22 schrieb Martin Gainty:
> > having difficulty locating ScheduledExecutorService in 
> > commons-nethttps://commons.apache.org/proper/commons-net/javadocs/api-3.5/index.htmlwhich
> >  version commons-net are you implementing?
> >
> > concerning ScheduledExecutorService running scheduleAtFixedRate or 
> > scheduleWithFixed methods there is a initial delay
> > built in so you maybe experiencing time-out after the initial connect 
> > during that initial delay
> > since we havent see the code you may want to look at 
> > ScheduledExecutorService  examples 
> > herehttps://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ScheduledExecutorService.html
> > Martin
> >> Subject: Re: NPE in getRemoteAdress
> >> To: user@commons.apache.org
> >> From: oliver.zem...@gmail.com
> >> Date: Mon, 29 Aug 2016 19:28:56 +0200
> >>
> >> My download method looks like this:
> >>
> >> private void downloadDirectory(FTPClient client, String sourceDir,
> >> String destDir,
> >>  ScheduledExecutorService
> >> scheduledExecutorService) throws IOException {
> >>
> >>   final FTPFile[] ftpFiles = client.listFiles(sourceDir);
> >>   for (final FTPFile ftpFile : ftpFiles) {
> >>
> >>   //skip . and ..
> >>   if (ftpFile.isDirectory() && (ftpFile.getName().equals(".")
> >> || ftpFile.getName().equals(".."))) {
> >>   continue;
> >>   }
> >>
> >>   if (ftpFile.isDirectory()) {
> >>   downloadDirectory(client, sourceDir +
> >> ftpFile.getName(), destDir + "/" + ftpFile.getName(),
> >> scheduledExecutorService);
> >>   } else {
> >> //scheduledExecutorService.submit(new
> >> DownloadRunnable(destDir, ftpFile, sourceDir, client));
> >>   FileOutputStream fos = new
> >> FileOutputStream(ftpFile.getName());
> >>   client.retrieveFile(sourceDir + ftpFile.getName(), fos );
> >>   fos.flush();
> >>   fos.close();
> >>   }
> >>   }
> >>   }
> >>
> >> When i run it like this, it works and it downloads the files. But when i
> >> use the scheduledExecuterService, it fails.
> >> So i replaced the 2 lines with my download method (where it uses the
> >> inputstream) to see what happens:
> >>
> >>} else {
> >> //scheduledExecutorService.submit(new
> >> DownloadRunnable(destDir, ftpFile, sourceDir, client));
> >>   new File(destDir).mkdirs();
> >>   final File outFile = new File(destDir, 
> >> ftpFile.getName());
> >>   final FileOutputStream fos = new 
> >> FileOutputStream(outFile);
> >>   final InputStream inputStream =
> >> client.retrieveFileStream(sourceDir + ftpFile.getName());
> >>
> >>   byte[] buffer = new byte[8096];
> >>   int len = -1;
> >>   try {
> >>   len = inputStream.read(buffer, 0, buffer.length);
> >>   } catch (Exception ex) {
> >>   ex.printStackTrace();
> 

RE: NPE in getRemoteAdress

[Martin Gainty]

having difficulty locating ScheduledExecutorService in 
commons-nethttps://commons.apache.org/proper/commons-net/javadocs/api-3.5/index.htmlwhich
 version commons-net are you implementing?

concerning ScheduledExecutorService running scheduleAtFixedRate or 
scheduleWithFixed methods there is a initial delay
built in so you maybe experiencing time-out after the initial connect during 
that initial delay
since we havent see the code you may want to look at ScheduledExecutorService  
examples 
herehttps://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ScheduledExecutorService.html
Martin
> Subject: Re: NPE in getRemoteAdress
> To: user@commons.apache.org
> From: oliver.zem...@gmail.com
> Date: Mon, 29 Aug 2016 19:28:56 +0200
> 
> My download method looks like this:
> 
> private void downloadDirectory(FTPClient client, String sourceDir, 
> String destDir,
> ScheduledExecutorService 
> scheduledExecutorService) throws IOException {
> 
>  final FTPFile[] ftpFiles = client.listFiles(sourceDir);
>  for (final FTPFile ftpFile : ftpFiles) {
> 
>  //skip . and ..
>  if (ftpFile.isDirectory() && (ftpFile.getName().equals(".") 
> || ftpFile.getName().equals(".."))) {
>  continue;
>  }
> 
>  if (ftpFile.isDirectory()) {
>  downloadDirectory(client, sourceDir + 
> ftpFile.getName(), destDir + "/" + ftpFile.getName(), 
> scheduledExecutorService);
>  } else {
> //scheduledExecutorService.submit(new 
> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>  FileOutputStream fos = new 
> FileOutputStream(ftpFile.getName());
>  client.retrieveFile(sourceDir + ftpFile.getName(), fos );
>  fos.flush();
>  fos.close();
>  }
>  }
>  }
> 
> When i run it like this, it works and it downloads the files. But when i 
> use the scheduledExecuterService, it fails.
> So i replaced the 2 lines with my download method (where it uses the 
> inputstream) to see what happens:
> 
>   } else {
> //scheduledExecutorService.submit(new 
> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>  new File(destDir).mkdirs();
>  final File outFile = new File(destDir, ftpFile.getName());
>  final FileOutputStream fos = new FileOutputStream(outFile);
>  final InputStream inputStream = 
> client.retrieveFileStream(sourceDir + ftpFile.getName());
> 
>  byte[] buffer = new byte[8096];
>  int len = -1;
>  try {
>  len = inputStream.read(buffer, 0, buffer.length);
>  } catch (Exception ex) {
>  ex.printStackTrace();
>  }
>  while (len != -1) {
>  fos.write(buffer, 0, len);
>  len = inputStream.read(buffer);
>  if (Thread.interrupted()) {
>  throw new InterruptedException();
>  }
>  }
>  client.completePendingCommand();
> //informListenersWeFinishedOneFile(sourceDir + 
> ftpFile.getName(), bytesWritten);
>  fos.flush();
>  fos.close();
>  }
>  }
> 
> That works too?!
> 
> I will go deeper into that at weekend and check whats going on, but at 
> the moment i really dont understand it. Maybe some kind of concurrency 
> problem with FTPClient?
> 
> 
> Am 29.08.2016 um 01:40 schrieb Martin Gainty:
> > possible timeout waiting for FTP to reply
> > examples.ftp.FTPClientExample says to increase FTP reply timeout with -w 
> > parameter
> >   if (args[base].equals("-w")) {
> > controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]);}
> > or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec 
> > reply timeout
> > ?
> > Martin
> > __
> >
> >
> >
> >> Date: Sun, 28 Aug 2016 20:06:46 +0200
> >> From: e...@zusammenkunft.net
> >> To: user@commons.apache.org
> >> CC: oliver.zem...@gmail.com
> >> Subject: Re: NPE in getRemoteAdress
> >>
> >> Hello,
> >>
> >> I am not sure about your NPE, but this code here ignores the result of
> >> the read call. It cannot deal with short reads:
> >>

Re: NPE in getRemoteAdress

[Oliver Zemann]

My download method looks like this:

private void downloadDirectory(FTPClient client, String sourceDir, 
String destDir,
   ScheduledExecutorService 
scheduledExecutorService) throws IOException {


final FTPFile[] ftpFiles = client.listFiles(sourceDir);
for (final FTPFile ftpFile : ftpFiles) {

//skip . and ..
if (ftpFile.isDirectory() && (ftpFile.getName().equals(".") 
|| ftpFile.getName().equals(".."))) {

continue;
}

if (ftpFile.isDirectory()) {
downloadDirectory(client, sourceDir + 
ftpFile.getName(), destDir + "/" + ftpFile.getName(), 
scheduledExecutorService);

} else {
//scheduledExecutorService.submit(new 
DownloadRunnable(destDir, ftpFile, sourceDir, client));
FileOutputStream fos = new 
FileOutputStream(ftpFile.getName());

client.retrieveFile(sourceDir + ftpFile.getName(), fos );
fos.flush();
fos.close();
}
}
}

When i run it like this, it works and it downloads the files. But when i 
use the scheduledExecuterService, it fails.
So i replaced the 2 lines with my download method (where it uses the 
inputstream) to see what happens:


 } else {
//scheduledExecutorService.submit(new 
DownloadRunnable(destDir, ftpFile, sourceDir, client));

new File(destDir).mkdirs();
final File outFile = new File(destDir, ftpFile.getName());
final FileOutputStream fos = new FileOutputStream(outFile);
final InputStream inputStream = 
client.retrieveFileStream(sourceDir + ftpFile.getName());


byte[] buffer = new byte[8096];
int len = -1;
try {
len = inputStream.read(buffer, 0, buffer.length);
} catch (Exception ex) {
ex.printStackTrace();
}
while (len != -1) {
fos.write(buffer, 0, len);
len = inputStream.read(buffer);
if (Thread.interrupted()) {
throw new InterruptedException();
}
}
client.completePendingCommand();
//informListenersWeFinishedOneFile(sourceDir + 
ftpFile.getName(), bytesWritten);

fos.flush();
fos.close();
}
}

That works too?!

I will go deeper into that at weekend and check whats going on, but at 
the moment i really dont understand it. Maybe some kind of concurrency 
problem with FTPClient?



Am 29.08.2016 um 01:40 schrieb Martin Gainty:

possible timeout waiting for FTP to reply
examples.ftp.FTPClientExample says to increase FTP reply timeout with -w 
parameter
  if (args[base].equals("-w")) {controlKeepAliveReplyTimeout = 
Integer.parseInt(args[++base]);}
or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec 
reply timeout
?
Martin
__




Date: Sun, 28 Aug 2016 20:06:46 +0200
From: e...@zusammenkunft.net
To: user@commons.apache.org
CC: oliver.zem...@gmail.com
Subject: Re: NPE in getRemoteAdress

Hello,

I am not sure about your NPE, but this code here ignores the result of
the read call. It cannot deal with short reads:

  Am Sun, 28 Aug 2016
15:50:36 +0200 schrieb Oliver Zemann <oliver.zem...@gmail.com>:

byte b[] =new byte[4096];
while (inputStream.read(b) != -1) {
   fos.write(b);
   bytesWritten += b.length;


Gruss
Bernd

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org






-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: NPE in getRemoteAdress

[Oliver Zemann]

Thanks Bernd, i will fix that.

Martin, i tried the following:

client.setDataTimeout(2);
client.setControlKeepAliveReplyTimeout(2);
client.setConnectTimeout(2);


Did not help, unfortunately.

Regards
Oli

Am 29.08.2016 um 01:40 schrieb Martin Gainty:

possible timeout waiting for FTP to reply
examples.ftp.FTPClientExample says to increase FTP reply timeout with -w 
parameter
  if (args[base].equals("-w")) {controlKeepAliveReplyTimeout = 
Integer.parseInt(args[++base]);}
or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec 
reply timeout
?
Martin
__




Date: Sun, 28 Aug 2016 20:06:46 +0200
From: e...@zusammenkunft.net
To: user@commons.apache.org
CC: oliver.zem...@gmail.com
Subject: Re: NPE in getRemoteAdress

Hello,

I am not sure about your NPE, but this code here ignores the result of
the read call. It cannot deal with short reads:

  Am Sun, 28 Aug 2016
15:50:36 +0200 schrieb Oliver Zemann <oliver.zem...@gmail.com>:

byte b[] =new byte[4096];
while (inputStream.read(b) != -1) {
   fos.write(b);
   bytesWritten += b.length;


Gruss
Bernd

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

RE: NPE in getRemoteAdress

[Martin Gainty]

possible timeout waiting for FTP to reply 
examples.ftp.FTPClientExample says to increase FTP reply timeout with -w 
parameter
 if (args[base].equals("-w")) {controlKeepAliveReplyTimeout = 
Integer.parseInt(args[++base]);}
or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec 
reply timeout
?
Martin 
__ 



> Date: Sun, 28 Aug 2016 20:06:46 +0200
> From: e...@zusammenkunft.net
> To: user@commons.apache.org
> CC: oliver.zem...@gmail.com
> Subject: Re: NPE in getRemoteAdress
> 
> Hello,
> 
> I am not sure about your NPE, but this code here ignores the result of
> the read call. It cannot deal with short reads:
> 
>  Am Sun, 28 Aug 2016
> 15:50:36 +0200 schrieb Oliver Zemann <oliver.zem...@gmail.com>:
> > byte b[] =new byte[4096];
> > while (inputStream.read(b) != -1) {
> >   fos.write(b);
> >   bytesWritten += b.length;
> 
> 
> Gruss
> Bernd
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
> For additional commands, e-mail: user-h...@commons.apache.org
> 
  

Re: NPE in getRemoteAdress

[Bernd Eckenfels]

Hello,

I am not sure about your NPE, but this code here ignores the result of
the read call. It cannot deal with short reads:

 Am Sun, 28 Aug 2016
15:50:36 +0200 schrieb Oliver Zemann :
> byte b[] =new byte[4096];
> while (inputStream.read(b) != -1) {
>   fos.write(b);
>   bytesWritten += b.length;


Gruss
Bernd

-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org

Re: NPE in getRemoteAdress

[Oliver Zemann]

ConnectionInfo is a JAXB Bean. I just have seen that thunderbird messed 
up some lines during copy/paste


The connectionInfo is fully initialized, here is a short part of it:

public class ConnectionInfo {

@XmlElement(name = "Name")
private String name;
@XmlElement(name = "Host")
private String host;
@XmlElement(name = "Port")
private int port;
@XmlElement(name = "Username")
private String username;
@XmlElement(name = "Password")
...

The connection to the FTP Server works. The FTP Client is logged in, CWD 
works, its just the transfer where i have problems.


Am 28.08.2016 um 16:55 schrieb Martin Gainty:

where is connectionInfo initialised?

-M


To: user@commons.apache.org
From: oliver.zem...@gmail.com
Subject: NPE in getRemoteAdress
Date: Sun, 28 Aug 2016 15:50:36 +0200

Hi

i am trying to get some files from a FTP Server. When i use
retrieveFile(), it works, but i would like to add some listeners to it
(how many bytes got transfered etc.) so i used retrieveFileStream().
Now i get some exceptions and no files are downloaded.

This is the part of the code:

@Override public void run() {

  try {
  new File(destDir).mkdirs();
  final File outFile =new File(destDir,ftpFile.getName());
  final FileOutputStream fos =new FileOutputStream(outFile);
  final InputStream inputStream =client.retrieveFileStream(sourceDir 
+ftpFile.getName());
  byte b[] =new byte[4096];//todo: would be nice to have such stuff in 
the appl conf so users can
config it long bytesWritten =0;
  while (inputStream.read(b) != -1) {
  fos.write(b);
  bytesWritten += b.length;
  informDetailListeners(sourceDir +ftpFile.getName(), bytesWritten);
  bytesTransferedTotal += b.length;
  informListeners(bytesTransferedTotal);
  }
  client.completePendingCommand();
  informListenersWeFinishedOneFile(sourceDir +ftpFile.getName(), 
bytesWritten);
  fos.flush();
  fos.close();
  System.out.println(client.getReplyString());
  }catch (Exception ex) {
  System.out.println(client.getReplyString());
  ex.printStackTrace();
  }
}

The NullpointerException occurs in retrieveFileStream() - when i step
into it i will get to FTPClient.java where it says:

final boolean isInet6Address =getRemoteAddress()instanceof Inet6Address;

getRemoteAddress() has this:return _socket_.getInetAddress();
When i try to debug into this, it just jumps into my catch block with a 
NullPointerException.
When i add this method as watch, it says socket not connected - but it must be 
connected ?!

The initialisation of the ftp Client is here:
private void downloadTo(String destinationDir)throws IOException {

  //first, create the destination directory so we can write into that final File file 
=new File(destinationDir +"/" +sfdlFile.getDescription());
  file.mkdirs();

  //set up the connection and go to the directory we want final 
ConnectionInfo connectionInfo =sfdlFile.getConnectionInfo();
  final String sourceDir =sfdlFile.getRemoteSourceDir();

  final FTPClient client =new FTPClient();

  client.connect(connectionInfo.getHost(), connectionInfo.getPort());
  System.out.println(client.getReplyString());
  client.enterLocalPassiveMode();
  System.out.println(client.getReplyString());
  client.login(connectionInfo.getUsername(), connectionInfo.getPassword());
  System.out.println(client.getReplyString());
  client.setFileType(FTP.BINARY_FILE_TYPE);
  System.out.println(client.getReplyString());
  client.changeWorkingDirectory(sourceDir);
  System.out.println(client.getReplyString());

  //now retrieve all files, recursive final ScheduledExecutorService 
scheduledExecutorService = 
Executors.newScheduledThreadPool(sfdlFile.getMaxThreads());
  downloadDirectory(client, sourceDir, destinationDir +"/" 
+sfdlFile.getDescription(), scheduledExecutorService);

  client.disconnect();
  System.out.println(client.getReplyString());
  //inform our downloader we finished }

The FTP Client is connected, the login worked, i get a lot of output from the 
Server which seems to be fine.
Can someone tell me whats wrong with that code? Is there something i did not 
read in handling recursive downloads of (bigger) files ?

The output in my intellij:
220 >>>>>>>>>>>>>>>>>>>>>>>>>

230 User logged in, proceed.

200 Type set to I.

250 Directory changed to /somedirectory

null
null
java.lang.NullPointerException
at 
org.apache.commons.net.SocketClient.getRemoteAddress(SocketClient.java:672)
at 
org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:819)
at 
org.apache.commons.net.ftp.FTPClient._retrieveFileStream(FTPClient.java:1984)
at 
org.apache.

RE: NPE in getRemoteAdress

[Martin Gainty]

where is connectionInfo initialised?

-M

> To: user@commons.apache.org
> From: oliver.zem...@gmail.com
> Subject: NPE in getRemoteAdress
> Date: Sun, 28 Aug 2016 15:50:36 +0200
> 
> Hi
> 
> i am trying to get some files from a FTP Server. When i use 
> retrieveFile(), it works, but i would like to add some listeners to it 
> (how many bytes got transfered etc.) so i used retrieveFileStream().
> Now i get some exceptions and no files are downloaded.
> 
> This is the part of the code:
> 
> @Override public void run() {
> 
>  try {
>  new File(destDir).mkdirs();
>  final File outFile =new File(destDir,ftpFile.getName());
>  final FileOutputStream fos =new FileOutputStream(outFile);
>  final InputStream inputStream =client.retrieveFileStream(sourceDir 
> +ftpFile.getName());
>  byte b[] =new byte[4096];//todo: would be nice to have such stuff in 
> the appl conf so users can 
> config it long bytesWritten =0;
>  while (inputStream.read(b) != -1) {
>  fos.write(b);
>  bytesWritten += b.length;
>  informDetailListeners(sourceDir +ftpFile.getName(), 
> bytesWritten);
>  bytesTransferedTotal += b.length;
>  informListeners(bytesTransferedTotal);
>  }
>  client.completePendingCommand();
>  informListenersWeFinishedOneFile(sourceDir +ftpFile.getName(), 
> bytesWritten);
>  fos.flush();
>  fos.close();
>  System.out.println(client.getReplyString());
>  }catch (Exception ex) {
>  System.out.println(client.getReplyString());
>  ex.printStackTrace();
>  }
> }
> 
> The NullpointerException occurs in retrieveFileStream() - when i step 
> into it i will get to FTPClient.java where it says:
> 
> final boolean isInet6Address =getRemoteAddress()instanceof Inet6Address;
> 
> getRemoteAddress() has this:return _socket_.getInetAddress();
> When i try to debug into this, it just jumps into my catch block with a 
> NullPointerException.
> When i add this method as watch, it says socket not connected - but it must 
> be connected ?!
> 
> The initialisation of the ftp Client is here:
> private void downloadTo(String destinationDir)throws IOException {
> 
>  //first, create the destination directory so we can write into that 
> final File file =new File(destinationDir +"/" +sfdlFile.getDescription());
>  file.mkdirs();
> 
>  //set up the connection and go to the directory we want final 
> ConnectionInfo connectionInfo =sfdlFile.getConnectionInfo();
>  final String sourceDir =sfdlFile.getRemoteSourceDir();
> 
>  final FTPClient client =new FTPClient();
> 
>  client.connect(connectionInfo.getHost(), connectionInfo.getPort());
>  System.out.println(client.getReplyString());
>  client.enterLocalPassiveMode();
>  System.out.println(client.getReplyString());
>  client.login(connectionInfo.getUsername(), connectionInfo.getPassword());
>  System.out.println(client.getReplyString());
>  client.setFileType(FTP.BINARY_FILE_TYPE);
>  System.out.println(client.getReplyString());
>  client.changeWorkingDirectory(sourceDir);
>  System.out.println(client.getReplyString());
> 
>  //now retrieve all files, recursive final ScheduledExecutorService 
> scheduledExecutorService = 
> Executors.newScheduledThreadPool(sfdlFile.getMaxThreads());
>  downloadDirectory(client, sourceDir, destinationDir +"/" 
> +sfdlFile.getDescription(), scheduledExecutorService);
> 
>  client.disconnect();
>  System.out.println(client.getReplyString());
>  //inform our downloader we finished }
> 
> The FTP Client is connected, the login worked, i get a lot of output from the 
> Server which seems to be fine.
> Can someone tell me whats wrong with that code? Is there something i did not 
> read in handling recursive downloads of (bigger) files ?
> 
> The output in my intellij:
> 220 >>>>>>>>>>>>>>>>>>>>>>>>>
> 
> 230 User logged in, proceed.
> 
> 200 Type set to I.
> 
> 250 Directory changed to /somedirectory
> 
> null
> null
> java.lang.NullPointerException
>   at 
> org.apache.commons.net.SocketClient.getRemoteAddress(SocketClient.java:672)
>   at 
> org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:819)
>   at 
> org.apache.commons.net.ftp.FTPClient._retrieveFileStream(FTPClient.java:1984)
>   at 
> org.apache.commons.net.ftp.FTPClient.retrieveFileStream(FTPClient.java:1971)
>   at 
> de.trustserv.jsfdl.app.

NPE in getRemoteAdress

[Oliver Zemann]

Hi

i am trying to get some files from a FTP Server. When i use 
retrieveFile(), it works, but i would like to add some listeners to it 
(how many bytes got transfered etc.) so i used retrieveFileStream().

Now i get some exceptions and no files are downloaded.

This is the part of the code:

@Override public void run() {

try {
new File(destDir).mkdirs();
final File outFile =new File(destDir,ftpFile.getName());
final FileOutputStream fos =new FileOutputStream(outFile);
final InputStream inputStream =client.retrieveFileStream(sourceDir 
+ftpFile.getName());
byte b[] =new byte[4096];//todo: would be nice to have such stuff in the appl conf so users can 
config it long bytesWritten =0;

while (inputStream.read(b) != -1) {
fos.write(b);
bytesWritten += b.length;
informDetailListeners(sourceDir +ftpFile.getName(), bytesWritten);
bytesTransferedTotal += b.length;
informListeners(bytesTransferedTotal);
}
client.completePendingCommand();
informListenersWeFinishedOneFile(sourceDir +ftpFile.getName(), 
bytesWritten);
fos.flush();
fos.close();
System.out.println(client.getReplyString());
}catch (Exception ex) {
System.out.println(client.getReplyString());
ex.printStackTrace();
}
}

The NullpointerException occurs in retrieveFileStream() - when i step 
into it i will get to FTPClient.java where it says:


final boolean isInet6Address =getRemoteAddress()instanceof Inet6Address;

getRemoteAddress() has this:return _socket_.getInetAddress();
When i try to debug into this, it just jumps into my catch block with a 
NullPointerException.
When i add this method as watch, it says socket not connected - but it must be 
connected ?!

The initialisation of the ftp Client is here:
private void downloadTo(String destinationDir)throws IOException {

//first, create the destination directory so we can write into that final File file 
=new File(destinationDir +"/" +sfdlFile.getDescription());
file.mkdirs();

//set up the connection and go to the directory we want final 
ConnectionInfo connectionInfo =sfdlFile.getConnectionInfo();
final String sourceDir =sfdlFile.getRemoteSourceDir();

final FTPClient client =new FTPClient();

client.connect(connectionInfo.getHost(), connectionInfo.getPort());
System.out.println(client.getReplyString());
client.enterLocalPassiveMode();
System.out.println(client.getReplyString());
client.login(connectionInfo.getUsername(), connectionInfo.getPassword());
System.out.println(client.getReplyString());
client.setFileType(FTP.BINARY_FILE_TYPE);
System.out.println(client.getReplyString());
client.changeWorkingDirectory(sourceDir);
System.out.println(client.getReplyString());

//now retrieve all files, recursive final ScheduledExecutorService 
scheduledExecutorService = 
Executors.newScheduledThreadPool(sfdlFile.getMaxThreads());
downloadDirectory(client, sourceDir, destinationDir +"/" 
+sfdlFile.getDescription(), scheduledExecutorService);

client.disconnect();
System.out.println(client.getReplyString());
//inform our downloader we finished }

The FTP Client is connected, the login worked, i get a lot of output from the 
Server which seems to be fine.
Can someone tell me whats wrong with that code? Is there something i did not 
read in handling recursive downloads of (bigger) files ?

The output in my intellij:
220 >

230 User logged in, proceed.

200 Type set to I.

250 Directory changed to /somedirectory

null
null
java.lang.NullPointerException
at 
org.apache.commons.net.SocketClient.getRemoteAddress(SocketClient.java:672)
at 
org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:819)
at 
org.apache.commons.net.ftp.FTPClient._retrieveFileStream(FTPClient.java:1984)
at 
org.apache.commons.net.ftp.FTPClient.retrieveFileStream(FTPClient.java:1971)
at 
de.trustserv.jsfdl.app.sfdl.SFDLFileDownloader$DownloadRunnable.run(SFDLFileDownloader.java:145)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)