Re: HdfsSpout

[K Zharas]

Yes, I have faced above mentioned problems after following that. Please,
reread my question.

On Wed, Feb 10, 2016 at 1:37 AM, Artem Ervits  wrote:

> does this help?
> https://github.com/apache/storm/tree/master/external/storm-hdfs
>
> On Tue, Feb 9, 2016 at 1:44 AM, K Zharas  wrote:
>
>> That is all about "HdfsBolt", not "HdfsSpout"
>>
>> On Tue, Feb 9, 2016 at 7:09 AM, Artem Ervits 
>> wrote:
>>
>>> Here is some info
>>> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_storm-user-guide/content/writing-data-with-storm-hdfs-connector.html
>>> On Feb 8, 2016 7:53 AM, "K Zharas"  wrote:
>>>
 Hi.

 1) How can I set "HdfsSpout" so that it will emit a tuple every X
 seconds?
   Is it done by "hdfsspout.commit.sec = 30"?
 2) I have only one file which has 300+ lines. After processing all the
 line it does not move it to "done" directory, and it still has ".lock" &
 ".inprogress". Also, it gives an error like "couldn't find the next file".
   What I want to do is to move the file into "done" directory,
 once "HdfsSpout" emits all the lines and to kill topology (stop program).

 Thank you.

>>>
>>
>>
>> --
>> Best regards,
>> Zharas
>>
>
>


-- 
Best regards,
Zharas

Re: HdfsSpout

[Artem Ervits]

does this help?
https://github.com/apache/storm/tree/master/external/storm-hdfs

On Tue, Feb 9, 2016 at 1:44 AM, K Zharas  wrote:

> That is all about "HdfsBolt", not "HdfsSpout"
>
> On Tue, Feb 9, 2016 at 7:09 AM, Artem Ervits 
> wrote:
>
>> Here is some info
>> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_storm-user-guide/content/writing-data-with-storm-hdfs-connector.html
>> On Feb 8, 2016 7:53 AM, "K Zharas"  wrote:
>>
>>> Hi.
>>>
>>> 1) How can I set "HdfsSpout" so that it will emit a tuple every X
>>> seconds?
>>>   Is it done by "hdfsspout.commit.sec = 30"?
>>> 2) I have only one file which has 300+ lines. After processing all the
>>> line it does not move it to "done" directory, and it still has ".lock" &
>>> ".inprogress". Also, it gives an error like "couldn't find the next file".
>>>   What I want to do is to move the file into "done" directory,
>>> once "HdfsSpout" emits all the lines and to kill topology (stop program).
>>>
>>> Thank you.
>>>
>>
>
>
> --
> Best regards,
> Zharas
>

HBase lookup bolt - exception

[Raja . Aravapalli]

Hi,

For the HBase lookup bolt of my storm topology, i am facing below exception... 
Request your help to identify and fix the issue ASAP.

java.lang.RuntimeException: java.lang.IllegalArgumentException: Row length is 0 
at 
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
 at 
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
 at 
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) 
at 
backtype.storm.daemon.executor$fn__5265$fn__5278$fn__5329.invoke(executor.clj:794)
 at backtype.storm.util$async_loop$fn__551.invoke(util.clj:465) at 
clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:744) 
Caused by: java.lang.IllegalArgumentException: Row length is 0 at 
org.apache.hadoop.hbase.client.Mutation.checkRow(Mutation.java:503) at 
org.apache.hadoop.hbase.client.Mutation.checkRow(Mutation.java:487) at 
org.apache.hadoop.hbase.client.Get.(Get.java:89) at 
org.apache.storm.hbase.common.HBaseClient.constructGetRequests(HBaseClient.java:112)
 at 
org.apache.storm.hbase.bolt.HBaseLookupBolt.execute(HBaseLookupBolt.java:65) at 
backtype.storm.daemon.executor$fn__5265$tuple_action_fn__5267.invoke(executor.clj:659)
 at 
backtype.storm.daemon.executor$mk_task_receiver$fn__5188.invoke(executor.clj:415)
 at 
backtype.storm.disruptor$clojure_handler$reify__1064.onEvent(disruptor.clj:58) 
at 
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
 ... 6 more


 Code used:

public class LookUpHBaseBolt {

public static HBaseLookupBolt getHBaseLookupBolt(String rkey, String 
colfam, String tablename) {

SimpleHBaseMapper lookupMapper = new 
SimpleHBaseMapper().withRowKeyField(rkey);

HBaseProjectionCriteria projectCriteria = new HBaseProjectionCriteria();
projectCriteria.addColumnFamily(colfam);

HBaseRowValueMapper row2TupleMapper = new HBaseRowValueMapper();

HBaseLookupBolt hbaseLkpBolt = new HBaseLookupBolt(tablename, 
lookupMapper, row2TupleMapper)
.withProjectionCriteria(projectCriteria);

return hbaseLkpBolt;
}

}

Please share your thoughts and help me fix the issue. Thanks.


Regards,
Raja.

Re: worker cannot download topology from nimbus host

[RAISUL BHUIYAN]

Hello Abhishek,
Thank you very much for your reply. It works fine when all the workers are in 
windows computer. But if I add Ubuntu then it starts to crashes. I checked the 
Nimbus log it does not shows any error report. I have added the ui_image and 
nimbus.log file.
Hope to hear from you./Raisul
 

On Tuesday, February 9, 2016 2:05 PM, Abhishek Agarwal 
 wrote:
 

 The logs suggest a different problem altogether. Supervisor is not able to 
connect to nimbus host. Check that nimbus is running and reachable from other 
machines. Make sure the nimbus.host in storm configuration is correct for all 
the supervisor machines
On Tue, Feb 9, 2016 at 4:19 PM, RAISUL BHUIYAN  wrote:

Hello Storm users,
I am working with Apache storm and created a cluster with two Windows-7 system 
and one Ubuntu-15.10 system. 
So the problem is whenever I run a topology I get java.lang.RuntimeException. 
it shows that Ubuntu worker cannot download the topology from windows nimbus 
host. However the other windows can download the topology. I changed the nimbus 
host to Ubuntu then the windows gives the same error. So I understand the error 
is with storm.local.directory. I havestorm.local.dir: 
"C:\\tmp\\storm_working_dir" (for windows) and 
storm.local.dir: "/usr/local/storm/data (for Ubuntu)
I was wondering if it is possible somehow to make the Ubuntu to be able to 
download from the windows host. I tried with sharing folder and everything but 
could not make it work.
I have attached the supervisor log error. Please help me with this issue. It 
will be a great favor to me.
Thank you very much in advance.
Regards,Raisul.





-- 
Regards,
Abhishek Agarwal


   

nimbus.log
Description: Binary data

Re: worker cannot download topology from nimbus host

[Abhishek Agarwal]

The logs suggest a different problem altogether. Supervisor is not able to
connect to nimbus host. Check that nimbus is running and reachable from
other machines. Make sure the nimbus.host in storm configuration is correct
for all the supervisor machines

On Tue, Feb 9, 2016 at 4:19 PM, RAISUL BHUIYAN 
wrote:

> Hello Storm users,
>
> I am working with Apache storm and created a cluster with two Windows-7
> system and one Ubuntu-15.10 system.
> So the problem is whenever I run a topology I get
> java.lang.RuntimeException. it shows that Ubuntu worker cannot download the
> topology from windows nimbus host. However the other windows can download
> the topology. I changed the nimbus host to Ubuntu then the windows gives
> the same error. So I understand the error is with storm.local.directory. I
> have
> storm.local.dir: "C:\\tmp\\storm_working_dir" (for windows) and
> storm.local.dir: "/usr/local/storm/data (for Ubuntu)
>
> I was wondering if it is possible somehow to make the Ubuntu to be able to
> download from the windows host. I tried with sharing folder and everything
> but could not make it work.
>
> I have attached the supervisor log error. Please help me with this issue.
> It will be a great favor to me.
>
> Thank you very much in advance.
>
> Regards,
> Raisul.
>
>


-- 
Regards,
Abhishek Agarwal

Re: The best way to unit test a Bolt with dependency?

[Spico Florin]

Hello!
  Set RabbitMQUtils as transient field in order to get rid of serialization
error. See if you can mock (Mockito using)  your RabbitMQ utils and inject
it via setMQ.
Hope that these help.
 Regards,
 Florin

On Tue, Jan 19, 2016 at 7:16 PM, Noppanit Charassinvichai <
noppani...@gmail.com> wrote:

> I'm trying to unit test my Bolt which has dependency to RabbitMQ.
>
> This is what I put in my prepare method.
>
> @Override
> public void prepare(Map map, TopologyContext topologyContext,
> OutputCollector outputCollector) {
> this.outputCollector = outputCollector;
> this.gson = new Gson();
> this.rabbitMQUtils = new RabbitMQUtils(this.config);
> }
>
> If I change that to inject RabbitMQUtils I get an Exception telling me
> that some classes in RabbitMQ API cannot be serialized.
>
> What do people do to unit test something like this?
>
> Thanks.
>

[Newbie Question] How to build DRPC based topology with multiple spouts

[Rahul R]

I have a topology which is similar to LinearDRPC topolgy mentioned in here
 . The major
difference being,  my bolts have multiple spouts. I am attaching a svg for
my topology as a visual aid.

Initially I had coded using localDRPC and things are pretty good.  The
following is my code

---
TopologyBuilder builder = new TopologyBuilder();
LocalDRPC drpc = new LocalDRPC();
DRPCSpout spout = new DRPCSpout("update_cache", drpc);
builder.setSpout("drpc", spout);
builder.setSpout("KakfaGeneratorSpout", new KafkaGeneratorSpout(), 1);


builder.setBolt("TransformBolt", new TransformationBolt(), 2)
.shuffleGrouping("KakfaGeneratorSpout")
.allGrouping("drpc");

builder.setBolt("return", new ReturnResults(),
1).noneGrouping("TransformBolt","drpc_stream");

builder.setBolt("NotificationBolt", new NotificationBolt() , 2)
.shuffleGrouping("TransformBolt");
--

I am not sure, how to port this remote storm nodes.  As a first step, I
tried building using LinearDRPCTopolgy Builder.  It seems to break most of
the things in existing code.  Any suggestions on how to go about building
this toplogy ?

Thanks,
./Rahul

PS:  I am running drpc server only on the nimbus host and not supervisors
hosts.