Re: Compression of Data in HDFS
Hi Bryan Thanks for your input, I did get it to work now. Sorry for the delayed response Just to confirm if it reads from a certain file and compresses and writes the compressed file to the Target Directory - how does nifi know that its has read from a certain file already? Or does it continue to read from Random Files? Thanks James -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Compression-of-Data-in-HDFS-tp8821p9061.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Compression of Data in HDFS
Hi Brian, Thanks for the help! I have tried two ways a. 1. I use GetHDFS to retrieve data from the HDFS , I then use putHDFS and set the compression to GZIP. 2. In the Directory I am putting the complete path i.e /landing/teradata/compressed/prodeiw_arc b. 1. I use GetHDFS to retrieve data from the HDFS, I then use Compress Content to apply the compression and then use PutHDFS 2. In the Directory I am putting the complete path i.e /landing/teradata/compressed/prodeiw_arc -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Compression-of-Data-in-HDFS-tp8821p8825.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Compression of Data in HDFS
I am trying to compress a whole lot of files from my HDFS and write to another folder on the HDFS My Folder Structure is as follows: \landing\databasename\prodeiw_arc\tablename\_SUCCESS \landing\databasename\prodeiw_arc\tablename\part-m-0 \landing\databasename\prodeiw_arc\tablename2\_SUCCESS \landing\databasename\prodeiw_arc\tablename2\part-m-0 I am trying to compress to the following \landing\compressed\prodeiw_arc\tablename\_SUCCESS \landing\compressed\prodeiw_arc\tablename\part-m-0 \landing\compressed\prodeiw_arc\tablename2\_SUCCESS \landing\compressed\prodeiw_arc\tablename2\part-m-0 I have found that it compresses to \landing\compressed\prodeiw_arc\_SUCCESS \landing\compressed\prodeiw_arc\tablename\part-m-0 it will then continue to overwrite. Is there anyway I can keep the directory structure when doing a PutHDFS? Thanks and Regards -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Compression-of-Data-in-HDFS-tp8821.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
