Storm-hdfs spout is not yet published in maven. You will have to checkout storm locally and build it to make it available for development.
From: K Zharas <kgzha...@gmail.com<mailto:kgzha...@gmail.com>> Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Date: Wednesday, February 3, 2016 at 11:41 AM To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Subject: Re: Storm + HDFS Yes, looks like it is. But, I have added dependencies required by storm-hdfs as stated in a guide. On Thu, Feb 4, 2016 at 3:33 AM, Nick R. Katsipoulakis <nick.kat...@gmail.com<mailto:nick.kat...@gmail.com>> wrote: Well, those errors look like a problem with the way you build your jar file. Please, make sure that you build your jar with the proper storm maven dependency). Cheers, Nick On Wed, Feb 3, 2016 at 2:31 PM, K Zharas <kgzha...@gmail.com<mailto:kgzha...@gmail.com>> wrote: It throws and error that packages does not exist. I have also tried changing org.apache to backtype, still got an error but only for storm.hdfs.spout. Btw, I use Storm-0.10.0 and Hadoop-2.7.1 package org.apache.storm does not exist package org.apache.storm does not exist package org.apache.storm.generated does not exist package org.apache.storm.metric does not exist package org.apache.storm.topology does not exist package org.apache.storm.utils does not exist package org.apache.storm.utils does not exist package org.apache.storm.hdfs.spout does not exist package org.apache.storm.hdfs.spout does not exist package org.apache.storm.topology.base does not exist package org.apache.storm.topology does not exist package org.apache.storm.tuple does not exist package org.apache.storm.task does not exist On Wed, Feb 3, 2016 at 8:57 PM, Matthias J. Sax <mj...@apache.org<mailto:mj...@apache.org>> wrote: Storm does provide HdfsSpout and HdfsBolt already. Just use those, instead of writing your own spout/bolt: https://github.com/apache/storm/tree/master/external/storm-hdfs -Matthias On 02/03/2016 12:34 PM, K Zharas wrote: > Can anyone help to create a Spout which reads a file from HDFS? > I have tried with the code below, but it is not working. > > public void nextTuple() { > Path pt=new Path("hdfs://localhost:50070/user/BCpredict.txt"); > FileSystem fs = FileSystem.get(new Configuration()); > BufferedReader br = new BufferedReader(new > InputStreamReader(fs.open(pt))); > String line = br.readLine(); > while (line != null){ > System.out.println(line); > line=br.readLine(); > _collector.emit(new Values(line)); > } > } > > On Tue, Feb 2, 2016 at 1:19 PM, K Zharas > <kgzha...@gmail.com<mailto:kgzha...@gmail.com> > <mailto:kgzha...@gmail.com<mailto:kgzha...@gmail.com>>> wrote: > > Hi. > > I have a project I'm currently working on. The idea is to implement > "scikit-learn" into Storm and integrate it with HDFS. > > I've already implemented "scikit-learn". But, currently I'm using a > text file to read and write. However, I need to use HDFS, but > finding it hard to integrate with HDFS. > > Here is the link to github > <https://github.com/kgzharas/StormTopologyTest>. (I only included > files that I used, not whole project) > > Basically, I have a few questions if you don't mint to answer them > 1) How to use HDFS to read and write? > 2) Is my "scikit-learn" implementation correct? > 3) How to create a Storm project? (Currently working in "storm-starter") > > These questions may sound a bit silly, but I really can't find a > proper solution. > > Thank you for your attention to this matter. > Sincerely, Zharas. > > > > > -- > Best regards, > Zharas -- Best regards, Zharas -- Nick R. Katsipoulakis, Department of Computer Science University of Pittsburgh -- Best regards, Zharas
