Har is transparent and hardly any performance overhead. You may decide not to compress or use a fast compression algorithm, such as snappy (recommended)
Le jeu. 3 sept. 2015 à 16:17, <nib...@free.fr> a écrit : > My main question in case of HAR usage is , is it possible to use Pig on it > and what about performances ? > > ----- Mail original ----- > De: "Jörn Franke" <jornfra...@gmail.com> > À: nib...@free.fr, user@spark.apache.org > Envoyé: Jeudi 3 Septembre 2015 15:54:42 > Objet: Re: Small File to HDFS > > > > > Store them as hadoop archive (har) > > > Le mer. 2 sept. 2015 à 18:07, < nib...@free.fr > a écrit : > > > Hello, > I'am currently using Spark Streaming to collect small messages (events) , > size being <50 KB , volume is high (several millions per day) and I have to > store those messages in HDFS. > I understood that storing small files can be problematic in HDFS , how can > I manage it ? > > Tks > Nicolas > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
