Thanks a lot, Akhil I saw this mail thread before, but still do not understand how to use XmlInputFormatof mahout in Spark Streaming (I am not Spark Streaming Expert yet ;-)). Can you show me some sample code for explanation.
Thanks in advance, Yong On Mon, Jun 22, 2015 at 6:44 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > You can use fileStream for that, look at the XMLInputFormat > <https://github.com/apache/mahout/blob/ad84344e4055b1e6adff5779339a33fa29e1265d/examples/src/main/java/org/apache/mahout/classifier/bayes/XmlInputFormat.java> > of mahout. It should give you full XML object as on record, (as opposed to > an XML record spread across multiple line records in textFileStream). Also > this > thread > <http://apache-spark-user-list.1001560.n3.nabble.com/Parsing-a-large-XML-file-using-Spark-td19239.html> > has some discussion around it. > > Thanks > Best Regards > > On Mon, Jun 22, 2015 at 12:23 AM, Yong Feng <fengyong...@gmail.com> wrote: > >> >> Hi Spark Experts >> >> I have a customer who wants to monitor coming data files (with xml >> format), and then analysize them after that put analysized data into DB. >> The size of each file is about 30MB (or even less in future). Spark >> streaming seems promising. >> >> After learning Spark Streaming and also google-ing how Spark Streaming >> handle xml files, I found there seems no existing Spark Stream utility to >> recognize whole xml file and parse it. The fileStream seems line-oriented. >> There is suggestion of putting whole xml file into one line, however it >> requires pre-processing files which will bring unexpected I/O. >> >> Can anyone throw some light on it? If will be great if there are some >> sample codes for me to start with. >> >> Thanks >> >> Yong >> >> >
