RE: Storm and enviroment variables

[Garcia-Contractor, Joseph (CORP)]

I did not see a –config option in the storm python start script… I did however 
find in there that if I set the STORM_CONF_DIR environment variable it would 
look for storm.yaml in that dir and this fixed my issue.

From: Priyank Shah [mailto:ps...@hortonworks.com]
Sent: Wednesday, October 07, 2015 5:31 PM
To: user@storm.apache.org
Subject: Re: Storm and enviroment variables

I think you can specify a config file that is other than the default. Can you 
try —config  command line option while running storm 
command ?

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org"
Date: Wednesday, October 7, 2015 at 2:15 PM
To: "user@storm.apache.org"
Subject: RE: Storm and enviroment variables

Ok so an additional question here…

Can I tell storm on startup to use a storm.yaml file that I specify with a full 
path, or will it only look in /etc/storm/conf/storm.yaml?

Joe

From: Priyank Shah [mailto:ps...@hortonworks.com]
Sent: Wednesday, September 30, 2015 2:31 PM
To: user@storm.apache.org
Subject: Re: Storm and enviroment variables

Hi Joseph,

I think storm core components don’t support environment variables. It either 
has to be a config yaml file or a –c option. For –c option, its a key value 
pair of the form key=value. I checked the code and key should be same as key in 
yaml. It should work for simple strings.  But not sure about other types. In 
the code its trying to parse it as a json. You can give it a try.

I don’t know about logback file . But you can configure your log directory 
using –c storm.log.dir=/var/log/ option.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "user@storm.apache.org"
Subject: Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of 
/etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir 
in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use –c at the command line but I would prefer using 
environment variable without having to build the command line args.  If I need 
to go with –c, what would the arguments still be in yaml format?  IE.: 
storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?

This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, notify the sender immediately by return email and delete the message 
and any attachments from your system.

Please help

[steve tueno]

Hi,
I'm implementing WordCount with storm. Sentences are read in a file and my
problem is what it the best way to stop LocalCluster when splout reach end
of file...

Cordialement,
TUENO FOTSO STEVE JEFFREY
Élève Ingénieur
5GI ENSP
+237 676 57 17 28
https://play.google.com/store/apps/details?id=com.polytech.remotecomputer
http://github.com/stuenofotso/remoteComputer
https://play.google.com/store/apps/details?id=com.polytech.androidsmssender
https://github.com/stuenofotso/notre-jargon

Re: Please help

[Ravi Sharma]

Hi Steve,
Storm 's basic design is to process stream(open ended or say No End) in
real time. There will be few hack ways of stoping the cluster once file is
finished but i guess none of them will be good looking.
Basically your storm cluster should be running all the time and waiting for
more message to process.

One dirty way is to send poison tuple after end of file, Bolt will receive
it, acknowledge it, and then when you receive ack in Spout you can kill ur
JVM manually. (Not good looking).

Ravi.

On Thu, Oct 8, 2015 at 5:17 PM, steve tueno  wrote:

> Hi,
> I'm implementing WordCount with storm. Sentences are read in a file and my
> problem is what it the best way to stop LocalCluster when splout reach end
> of file...
>
> Cordialement,
> TUENO FOTSO STEVE JEFFREY
> Élève Ingénieur
> 5GI ENSP
> +237 676 57 17 28
> https://play.google.com/store/apps/details?id=com.polytech.remotecomputer
> http://github.com/stuenofotso/remoteComputer
> https://play.google.com/store/apps/details?id=com.polytech.androidsmssender
>
> https://github.com/stuenofotso/notre-jargon
>

Re: Storm and enviroment variables

[Priyank Shah]

Glad that it fixed your issue. The python script has the —config option. Check 
https://github.com/apache/storm/blob/master/bin/storm.py#L632 out.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org"
Date: Thursday, October 8, 2015 at 6:52 AM
To: "user@storm.apache.org"
Subject: RE: Storm and enviroment variables

I did not see a –config option in the storm python start script… I did however 
find in there that if I set the STORM_CONF_DIR environment variable it would 
look for storm.yaml in that dir and this fixed my issue.

From: Priyank Shah [mailto:ps...@hortonworks.com]
Sent: Wednesday, October 07, 2015 5:31 PM
To: user@storm.apache.org
Subject: Re: Storm and enviroment variables

I think you can specify a config file that is other than the default. Can you 
try —config  command line option while running storm 
command ?

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org"
Date: Wednesday, October 7, 2015 at 2:15 PM
To: "user@storm.apache.org"
Subject: RE: Storm and enviroment variables

Ok so an additional question here…

Can I tell storm on startup to use a storm.yaml file that I specify with a full 
path, or will it only look in /etc/storm/conf/storm.yaml?

Joe

From: Priyank Shah [mailto:ps...@hortonworks.com]
Sent: Wednesday, September 30, 2015 2:31 PM
To: user@storm.apache.org
Subject: Re: Storm and enviroment variables

Hi Joseph,

I think storm core components don’t support environment variables. It either 
has to be a config yaml file or a –c option. For –c option, its a key value 
pair of the form key=value. I checked the code and key should be same as key in 
yaml. It should work for simple strings.  But not sure about other types. In 
the code its trying to parse it as a json. You can give it a try.

I don’t know about logback file . But you can configure your log directory 
using –c storm.log.dir=/var/log/ option.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "user@storm.apache.org"
Subject: Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of 
/etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir 
in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use –c at the command line but I would prefer using 
environment variable without having to build the command line args.  If I need 
to go with –c, what would the arguments still be in yaml format?  IE.: 
storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?

This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, notify the sender immediately by return email and delete the message 
and any attachments from your system.

Help Required

[Rudraneel chakraborty]

I wanted to know , if there are any best practice guidelines while
designing topology. For example, is it a good idea to introduce multiple
topologies by breaking a big data processing task into stages? Is there any
helpful tutorial or reference which I can use to learn about dividing tasks
among topologies?

-- 
Rudraneel Chakraborty
Carleton University Real Time and Distributed Systems Reserach

Re: Please help

[steve tueno]

thanks you very much

Cordialement,
TUENO FOTSO STEVE JEFFREY
Élève Ingénieur
5GI ENSP
+237 676 57 17 28
https://play.google.com/store/apps/details?id=com.polytech.remotecomputer
http://github.com/stuenofotso/remoteComputer
https://play.google.com/store/apps/details?id=com.polytech.androidsmssender
https://github.com/stuenofotso/notre-jargon

2015-10-08 17:37 GMT+01:00 Ravi Sharma :

> Hi Steve,
> Storm 's basic design is to process stream(open ended or say No End) in
> real time. There will be few hack ways of stoping the cluster once file is
> finished but i guess none of them will be good looking.
> Basically your storm cluster should be running all the time and waiting
> for more message to process.
>
> One dirty way is to send poison tuple after end of file, Bolt will receive
> it, acknowledge it, and then when you receive ack in Spout you can kill ur
> JVM manually. (Not good looking).
>
> Ravi.
>
> On Thu, Oct 8, 2015 at 5:17 PM, steve tueno  wrote:
>
>> Hi,
>> I'm implementing WordCount with storm. Sentences are read in a file and
>> my problem is what it the best way to stop LocalCluster when splout reach
>> end of file...
>>
>> Cordialement,
>> TUENO FOTSO STEVE JEFFREY
>> Élève Ingénieur
>> 5GI ENSP
>> +237 676 57 17 28
>> https://play.google.com/store/apps/details?id=com.polytech.remotecomputer
>> http://github.com/stuenofotso/remoteComputer
>>
>> https://play.google.com/store/apps/details?id=com.polytech.androidsmssender
>>
>> https://github.com/stuenofotso/notre-jargon
>>
>
>

RE: Storm and enviroment variables

[Garcia-Contractor, Joseph (CORP)]

Yeap… it certainly does… I missed that.  Thanks for pointing it out!   ☺


From: Priyank Shah [mailto:ps...@hortonworks.com]
Sent: Thursday, October 08, 2015 12:51 PM
To: user@storm.apache.org
Subject: Re: Storm and enviroment variables

Glad that it fixed your issue. The python script has the —config option. Check 
https://github.com/apache/storm/blob/master/bin/storm.py#L632 out.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org"
Date: Thursday, October 8, 2015 at 6:52 AM
To: "user@storm.apache.org"
Subject: RE: Storm and enviroment variables

I did not see a –config option in the storm python start script… I did however 
find in there that if I set the STORM_CONF_DIR environment variable it would 
look for storm.yaml in that dir and this fixed my issue.

From: Priyank Shah [mailto:ps...@hortonworks.com]
Sent: Wednesday, October 07, 2015 5:31 PM
To: user@storm.apache.org
Subject: Re: Storm and enviroment variables

I think you can specify a config file that is other than the default. Can you 
try —config  command line option while running storm 
command ?

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org"
Date: Wednesday, October 7, 2015 at 2:15 PM
To: "user@storm.apache.org"
Subject: RE: Storm and enviroment variables

Ok so an additional question here…

Can I tell storm on startup to use a storm.yaml file that I specify with a full 
path, or will it only look in /etc/storm/conf/storm.yaml?

Joe

From: Priyank Shah [mailto:ps...@hortonworks.com]
Sent: Wednesday, September 30, 2015 2:31 PM
To: user@storm.apache.org
Subject: Re: Storm and enviroment variables

Hi Joseph,

I think storm core components don’t support environment variables. It either 
has to be a config yaml file or a –c option. For –c option, its a key value 
pair of the form key=value. I checked the code and key should be same as key in 
yaml. It should work for simple strings.  But not sure about other types. In 
the code its trying to parse it as a json. You can give it a try.

I don’t know about logback file . But you can configure your log directory 
using –c storm.log.dir=/var/log/ option.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "user@storm.apache.org"
Subject: Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of 
/etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir 
in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use –c at the command line but I would prefer using 
environment variable without having to build the command line args.  If I need 
to go with –c, what would the arguments still be in yaml format?  IE.: 
storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?

This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, notify the sender immediately by return email and delete the message 
and any attachments from your system.