Re: New Distributed Cache

2014-01-09 Thread Bill Q 
Hi Omkar,
Did you find out why the getLocalCacheFiles is deprecated? If it is indeed
deprecated, what would be the alternative to use?

On Thursday, July 11, 2013, Omkar Joshi wrote:

 Yeah Andrew.. there seems to be some problem with context.getCacheFiles()
 api which is returning null..

  Path[] cachedFilePaths =

   context.getLocalCacheFiles(); // I am checking why it is
 deprecated...

   for (Path cachedFilePath : cachedFilePaths) {

 File cachedFile = new File(cachedFilePath.toUri().getRawPath());

 System.out.println(cached fie path  

 + cachedFile.getAbsolutePath());

   }

 I hope this helps for the time being.. JobContext was suppose to replace
 DistributedCache api (it will be deprecated) however there is some problem
 with that or I am missing something... Will reply if I find the solution to
 it.

 context.getCacheFiles will give you the uri used for localizing files...
 (original uri used for adding it to cache)... However you can use
 DistributedCache.getCacheFiles() api till context api is fixed.

 context.getLocalCacheFiles .. will give you the actual file path on node
 manager... (after file is localized).

 Thanks,
 Omkar Joshi
 *Hortonworks Inc.* http://www.hortonworks.com


 On Thu, Jul 11, 2013 at 8:19 AM, Botelho, Andrew 
 andrew.bote...@emc.comwrote:

 So in my driver code, I try to store the file in the cache with this line
 of code:



 job.addCacheFile(new URI(file location));



 Then in my Mapper code, I do this to try and access the cached file:



 URI[] localPaths = context.getCacheFiles();

 File f = new File(localPaths[0]);



 However, I get a NullPointerException when I do that in the Mapper code.



 Any suggesstions?



 Andrew



 *From:* Shahab Yunus [mailto:shahab.yu...@gmail.com]
 *Sent:* Wednesday, July 10, 2013 9:43 PM
 *To:* user@hadoop.apache.org
 *Subject:* Re: New Distributed Cache



 Also, once you have the array of URIs after calling getCacheFiles  you
 can iterate over them using File class or Path (
 http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/Path.html#Path(java.net.URI)
 )



 Regards,

 Shahab



 On Wed, Jul 10, 2013 at 5:08 PM, Omkar Joshi ojo...@hortonworks.com
 wrote:

 did you try JobContext.getCacheFiles() ?




 Thanks,

 Omkar Joshi

 http://www.hortonworks.com



-- 
Many thanks.


Bill

RE: New Distributed Cache

2013-07-11 Thread Botelho, Andrew 
So in my driver code, I try to store the file in the cache with this line of 
code:

job.addCacheFile(new URI(file location));

Then in my Mapper code, I do this to try and access the cached file:

URI[] localPaths = context.getCacheFiles();
File f = new File(localPaths[0]);

However, I get a NullPointerException when I do that in the Mapper code.

Any suggesstions?

Andrew

From: Shahab Yunus [mailto:shahab.yu...@gmail.com]
Sent: Wednesday, July 10, 2013 9:43 PM
To: user@hadoop.apache.org
Subject: Re: New Distributed Cache

Also, once you have the array of URIs after calling getCacheFiles  you can 
iterate over them using File class or Path 
(http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/Path.html#Path(java.net.URI))

Regards,
Shahab

On Wed, Jul 10, 2013 at 5:08 PM, Omkar Joshi 
ojo...@hortonworks.commailto:ojo...@hortonworks.com wrote:
did you try JobContext.getCacheFiles() ?


Thanks,
Omkar Joshi
Hortonworks Inc.http://www.hortonworks.com

On Wed, Jul 10, 2013 at 10:15 AM, Botelho, Andrew 
andrew.bote...@emc.commailto:andrew.bote...@emc.com wrote:
Hi,

I am trying to store a file in the Distributed Cache during my Hadoop job.
In the driver class, I tell the job to store the file in the cache with this 
code:

Job job = Job.getInstance();
job.addCacheFile(new URI(file name));

That all compiles fine.  In the Mapper code, I try accessing the cached file 
with this method:

Path[] localPaths = context.getLocalCacheFiles();

However, I am getting warnings that this method is deprecated.
Does anyone know the newest way to access cached files in the Mapper code? (I 
am using Hadoop 2.0.5)

Thanks in advance,

Andrew

Re: New Distributed Cache

2013-07-11 Thread Omkar Joshi 
Yeah Andrew.. there seems to be some problem with context.getCacheFiles()
api which is returning null..

 Path[] cachedFilePaths =

  context.getLocalCacheFiles(); // I am checking why it is
deprecated...

  for (Path cachedFilePath : cachedFilePaths) {

File cachedFile = new File(cachedFilePath.toUri().getRawPath());

System.out.println(cached fie path  

+ cachedFile.getAbsolutePath());

  }

I hope this helps for the time being.. JobContext was suppose to replace
DistributedCache api (it will be deprecated) however there is some problem
with that or I am missing something... Will reply if I find the solution to
it.

context.getCacheFiles will give you the uri used for localizing files...
(original uri used for adding it to cache)... However you can use
DistributedCache.getCacheFiles() api till context api is fixed.

context.getLocalCacheFiles .. will give you the actual file path on node
manager... (after file is localized).

Thanks,
Omkar Joshi
*Hortonworks Inc.* http://www.hortonworks.com


On Thu, Jul 11, 2013 at 8:19 AM, Botelho, Andrew andrew.bote...@emc.comwrote:

 So in my driver code, I try to store the file in the cache with this line
 of code:

 ** **

 job.addCacheFile(new URI(file location));

 ** **

 Then in my Mapper code, I do this to try and access the cached file:

 ** **

 URI[] localPaths = context.getCacheFiles();

 File f = new File(localPaths[0]);

 ** **

 However, I get a NullPointerException when I do that in the Mapper code.**
 **

 ** **

 Any suggesstions?

 ** **

 Andrew

 ** **

 *From:* Shahab Yunus [mailto:shahab.yu...@gmail.com]
 *Sent:* Wednesday, July 10, 2013 9:43 PM
 *To:* user@hadoop.apache.org
 *Subject:* Re: New Distributed Cache

 ** **

 Also, once you have the array of URIs after calling getCacheFiles  you
 can iterate over them using File class or Path (
 http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/Path.html#Path(java.net.URI)
 )

 ** **

 Regards,

 Shahab

 ** **

 On Wed, Jul 10, 2013 at 5:08 PM, Omkar Joshi ojo...@hortonworks.com
 wrote:

 did you try JobContext.getCacheFiles() ?

 ** **


 

 Thanks,

 Omkar Joshi

 *Hortonworks Inc.* http://www.hortonworks.com

 ** **

 On Wed, Jul 10, 2013 at 10:15 AM, Botelho, Andrew andrew.bote...@emc.com
 wrote:

 Hi,

  

 I am trying to store a file in the Distributed Cache during my Hadoop job.
 

 In the driver class, I tell the job to store the file in the cache with
 this code:

  

 Job job = Job.getInstance();

 job.addCacheFile(new URI(file name));

  

 That all compiles fine.  In the Mapper code, I try accessing the cached
 file with this method:

  

 Path[] localPaths = context.getLocalCacheFiles();

  

 However, I am getting warnings that this method is deprecated.

 Does anyone know the newest way to access cached files in the Mapper code?
 (I am using Hadoop 2.0.5)

  

 Thanks in advance,

  

 Andrew

 ** **

 ** **

New Distributed Cache

2013-07-10 Thread Botelho, Andrew 
Hi,

I am trying to store a file in the Distributed Cache during my Hadoop job.
In the driver class, I tell the job to store the file in the cache with this 
code:

Job job = Job.getInstance();
job.addCacheFile(new URI(file name));

That all compiles fine.  In the Mapper code, I try accessing the cached file 
with this method:

Path[] localPaths = context.getLocalCacheFiles();

However, I am getting warnings that this method is deprecated.
Does anyone know the newest way to access cached files in the Mapper code? (I 
am using Hadoop 2.0.5)

Thanks in advance,

Andrew

Re: New Distributed Cache

2013-07-10 Thread Omkar Joshi 
did you try JobContext.getCacheFiles() ?


Thanks,
Omkar Joshi
*Hortonworks Inc.* http://www.hortonworks.com


On Wed, Jul 10, 2013 at 10:15 AM, Botelho, Andrew andrew.bote...@emc.comwrote:

 Hi,

 ** **

 I am trying to store a file in the Distributed Cache during my Hadoop job.
 

 In the driver class, I tell the job to store the file in the cache with
 this code:

 ** **

 Job job = Job.getInstance();

 job.addCacheFile(new URI(file name));

 ** **

 That all compiles fine.  In the Mapper code, I try accessing the cached
 file with this method:

 ** **

 Path[] localPaths = context.getLocalCacheFiles();

 ** **

 However, I am getting warnings that this method is deprecated.

 Does anyone know the newest way to access cached files in the Mapper code?
 (I am using Hadoop 2.0.5)

 ** **

 Thanks in advance,

 ** **

 Andrew

Re: New Distributed Cache

2013-07-10 Thread Shahab Yunus 
Also, once you have the array of URIs after calling getCacheFiles  you can
iterate over them using File class or Path (
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/Path.html#Path(java.net.URI)
)

Regards,
Shahab


On Wed, Jul 10, 2013 at 5:08 PM, Omkar Joshi ojo...@hortonworks.com wrote:

 did you try JobContext.getCacheFiles() ?


 Thanks,
 Omkar Joshi
 *Hortonworks Inc.* http://www.hortonworks.com


 On Wed, Jul 10, 2013 at 10:15 AM, Botelho, Andrew 
 andrew.bote...@emc.comwrote:

 Hi,

 ** **

 I am trying to store a file in the Distributed Cache during my Hadoop job.
 

 In the driver class, I tell the job to store the file in the cache with
 this code:

 ** **

 Job job = Job.getInstance();

 job.addCacheFile(new URI(file name));

 ** **

 That all compiles fine.  In the Mapper code, I try accessing the cached
 file with this method:

 ** **

 Path[] localPaths = context.getLocalCacheFiles();

 ** **

 However, I am getting warnings that this method is deprecated.

 Does anyone know the newest way to access cached files in the Mapper
 code? (I am using Hadoop 2.0.5)

 ** **

 Thanks in advance,

 ** **

 Andrew