Re: Why tuples fail in spout

but no failed count is shown for bolts *Best regards,* *K.Sai Dilip Reddy.* On Tue, May 3, 2016 at 11:19 AM, John Fang wrote: > Some tuples failed in the bolts. You can review the bolts' code. Maybe > your bolts' code trigger the fail() due to some reasons, Or the operation > of bolts need m

Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm

Harsha, Yes sure, I applied regex for filtering by metric name. Please refer relevant unit test from pull request . I'm c

回复：Why tuples fail in spout

Some tuples failed in the bolts. You can review the bolts' code. Maybe your bolts' code trigger the fail() due to some reasons, Or the operation of bolts need more time.  --发件人：Sai Dilip Reddy Kiralam 发送时间：2016年5月3日(星期二) 12:06收件人：us

Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm

Jungtaek, I think filters that can support a regex gives more felxibility. Thanks, Harsha On Mon, May 2, 2016, at 07:48 PM, Jungtaek Lim wrote: > Kevin, > > For specific task, you can register your own metrics which resides > per task. > But metrics doc on Storm is not kind enough to let users

Why tuples fail in spout

Hi all, I'm running storm topology in the local machine & storm UI shows some tuples are failed in the spout.as per my knowledge spout tuples are transferred to a bolts with out any failure.Can any of you help me out in finding the reason of tuples failures in the spout. *Best regards,* *K

Re: Configuring Python environment

I did try streamparse, but it failed to deploy the topology to the remote Storm cluster because of some internal errors which I wasn't able to track down. I am not surprised though, because even manual install of python 2.7 or virtual environment is painful on CentOS 6.5. So, I took the topology JA

Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm

Kevin, For specific task, you can register your own metrics which resides per task. But metrics doc on Storm is not kind enough to let users follow, so I addressed this and submitted pull request. https://github.com/HeartSaVioR/storm/blob/STORM-1724-1.x/docs/Metrics.md There're no custom worker m

Re: Configuring Python environment

You can try streamparse, I haven't used it, but it looks charming. On Tuesday, May 3, 2016, Alec Swan wrote: > Thanks, Joaquin, but I've seen all these articles since I spent the entire > weekend trying to get my topology to run on CentOS/Python2.7. > Getting it to work on Mac OSX was a breeze b

Re: Configuring Python environment

Thanks, Joaquin, but I've seen all these articles since I spent the entire weekend trying to get my topology to run on CentOS/Python2.7. Getting it to work on Mac OSX was a breeze because I can easily install Python 2.7 on it. It'd be great if somebody with the actually CentOS/Python2.7 experience

Re: Configuring Python environment

virtualenvwrappers - http://virtualenvwrapper.readthedocs.io/en/latest/ - https://pypi.python.org/pypi/virtualenvwrapper/ Pyenv - https://amaral.northwestern.edu/resources/guides/pyenv-tutorial - https://github.com/yyuu/pyenv Docker You can have a Python in another directory, then put that d

Configuring Python environment

Hello, I am having a real hard time configuring Storm to use python2.7 virtual environment that I installed on my CentOS 6.5 host. CentOS 6.5 ships with python2.6 but storm requires at least python2.7. Switching to python2.7 breaks some of CentOS functionality, so I had to install python2.7 as a

Re: Spout Questions

Thank you for the feedback. I've juste switched back to 0.10 and I don't have the issue. nextTuple is quite fast, I'm handleling a buffer of Tuple, when the buffer is empty, I'm loading data from redis. Ack method is deleting data on redis using pipeline to be fast. It's strange that in the 0.1

Re: Spout Questions

nextTuple(), ack(), and fail() are all called by the same thread. nextTuple() should be fast, so you probably only want to emit one or a handful of tuples. Emitting a huge number of tuples in the nextTuple() method is what’s causing your problem. -Taylor > On May 2, 2016, at 9:08 AM, Adrien Ca

Storm-Vagrant for 0.10.0

I was able to get a successful setup for Apache Storm 0.10.0 using Vagrant. Thanks for the help. I used some of the same scripts to build out one in AWS as well. - https://github.com/darkn3rd/storm-vagrant -- 是故勝兵先勝而後求戰，敗兵先戰而後求勝。

Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm

Jungtaek, Probably a filter config to whitelist and blacklist certain metrics. So that it will scale if there are too many workers and users can turn off certain metrics. Thanks, Harsha On Mon, May 2, 2016, at 06:19 AM, Stephen Powis wrote: > Oooh I'd love this as well!  I really dig the ease

Re: How Does Nimbus Decide to Restart Topology?

Some more interesting info. As I mentioned in the last post, we have 6 workers. Each worker has 2 slots. Only one of the supervisors received a state :timed-out message. After it received the message, it restarted both workers successfully. The other 5 supervisors all experienced the same issu

Re: How Does Nimbus Decide to Restart Topology?

Unfortunately we're not capturing disk i/o in our metrics, I can look in to doing that for next time. We're not capturing GC logs, we are using the graphite storm metric consumer to push metrics to graphite, one of which is the GC time from the default GC mxbean. > I'm assuming you're saying that

Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm

Oooh I'd love this as well! I really dig the ease of the metric framework in storm and have all the metrics go thru one centralized config. But as the number of storm hosts and number of tasks grow, I've found that Graphite/Grafana has a hard time collecting up all the relevant metrics across a l

Spout Questions

Hi there, Dont't know if I'm on the right place.. But let's try. I'm build a Topology, And I've a spout plugged on Redis. My question is, when the topology is active, Why the nextTuple() method isn't call when ack() method is called. Meaning, I've about 10k acking message without a nextTuple()

Re: Is Storm 1.0.0 compatible with Kafka 0.8.2.x?

good thing is that storm-kafka artifact itself is compatible with kafka 0.8.2.1. So in your maven project, you can simply exclude the org.apache.kafka dependencies coming out of storm-kafka project and use the version you need. On Mon, May 2, 2016 at 3:03 PM, Abhishek Agarwal wrote: > John, I th

Re: Firewall - what ports should be open?

You may want to open up logviewer ports as well (on each supervisor machine). If you are using drpc (optional), drpc server port should be opened up as well. On Mon, May 2, 2016 at 2:11 PM, Joaquin Menchaca wrote: > In AWS land, everything is locked off unless opened with SG. I > enabled standa

Re: UI showing no topologies

Ignore the last part. There was a permission problem on local.dir, no idea how ownership changed on that... weird. On Mon, May 2, 2016 at 2:33 AM, Joaquin Menchaca wrote: > I was doing originally: > > $ storm jar storm-starter-0.10.0.jar storm.starter.ExclamationTopology > > But finding some d

Re: Is Storm 1.0.0 compatible with Kafka 0.8.2.x?

John, I think you have hit it right. I started using storm-kafka 1.0 with kafka server 0.8.2.1 and started running into all sorts of issues including the one you pointed out. Also 0.9 documentation clearly states that upgraded clients will not be compatible with older kafka version. http://kafka.ap

Re: UI showing no topologies

I was doing originally: $ storm jar storm-starter-0.10.0.jar storm.starter.ExclamationTopology But finding some docs on HortonNetworks, I ran it with a parameter, which it uses to register a topology name, I guess: $ storm jar /vagrant/topologies/storm-starter-0.10.0.jar storm.starter.Exclamatio

Re: UI showing no topologies

In your topology code, what call are you using to deploy to storm? Best Regards Matthew Lowe > On 02 May 2016, at 10:43, Joaquin Menchaca wrote: > > But I seem to be running topologies only on nimbus. > > The ui is not showing anything under Topology Summary. It says "No > data available in

Any Docs on Topologies (more than Readme)

The readme mentions options, for things like parameters of "production-topology remote" for RollingTopWords. But the information is sparse, there's little to no information on these topologies. Are there any docs anywhere on them, some of their options, etc.? Other than going through the code?

UI showing no topologies

But I seem to be running topologies only on nimbus. The ui is not showing anything under Topology Summary. It says "No data available in table". It does show three systems in Supervisor Summary. I ran using storm starter topologies: "RollingTopWords", "WordCountTopology", and "ExclaimationTop

Re: id for bolt tasks in Storm UI

Hi Yury! Thanks for your reply, now it's clear to me. 2016-04-29 14:46 GMT+02:00 Yury Ruchin : > Hi there Serega, > > What you see in the UI are probably compound executor "ID"s. They are > actually ranges of task IDs assigned to respective executors. For example, > [26-27] means executor with ta

Firewall - what ports should be open?

In AWS land, everything is locked off unless opened with SG. I enabled standard supervisor ports for supervisor (6700-6703), enabled default storm nimbus port (tcp/6627), and default ui (tcp/8080), and of course have healthy zookeeper (tcp/2181). Does anything else need to be opened? -- 是故勝兵先勝

Re: Cannot launch Supervisor, missing unknown file

I bring up another scratch system to see if I can reproduce it. Unfamiliar with Python, what would I need to do? I installed Java 1.8, then ran 'bin/storm supervisor &' from storm home directory. On Sun, May 1, 2016 at 11:47 PM, Erik Weathers wrote: > Do you know what precise package version of