details are already documented.
3. I am also looking to make this feature available in older versions
starting from 1.x.x till latest. Let me know what is the procedure for
that. AFAIS, the flux core got repackaged between this time frame and
perhaps I may have to create a separate branch to take ca
Sure, will look to submit a draft PR this weekend.
On Thu, 15 Jun 2023, 12:20 Richard Zowalla, wrote:
> Hey,
>
> I think, that it would be a good addition.
> If you have some spare time, I guess, that a PR would be very welcome.
>
> Gruß
> Richard
>
>
> Am 15. Juni 2023 08:16:14 MESZ schrieb 6ha
Hey,
I think, that it would be a good addition.
If you have some spare time, I guess, that a PR would be very welcome.
Gruß
Richard
Am 15. Juni 2023 08:16:14 MESZ schrieb 6harat :
>Hey,
>
>A while back the support for declaring WorkerHook in TopologyBuilder was
>added. Related JIRA: https://iss
Hey,
A while back the support for declaring WorkerHook in TopologyBuilder was
added. Related JIRA: https://issues.apache.org/jira/browse/STORM-126
However, this feature has still not made it to FluxBuilder. Wanted to check
if there are plans to bring parity in this regard.
Seems like a small cha
Hi,
My name is Paul. Previously I was using storm 0.10.0 and now have to upgrade to
the latest 2.2.0.
I am using flux. But it isn't working. This is my config:
name: "AuditLogConsumerTopology"
components:
- id: "kafkaConfigBu
Hi,
My name is Paul. Previously I was using storm 0.10.0 and now have to upgrade to
the latest 2.2.0.
I am using flux. But it isn't working. This is my config:
name: "AuditLogConsumerTopology"
components:
- id: "kafkaConfigBu
I am not sure if this can be achieved via flux or not although Storm 2.0
introduced support for lambda expressions.
https://issues.apache.org/jira/browse/STORM-2490
I think, Taylor is the original author of flux. If there is a way to access
this feature via flux... he might be able to tell
Anyone have info/tips?
On Wednesday, December 4, 2019, 11:08:36 AM EST, joe smith
wrote:
Hi,
Would like to know if there is a way to write a lambda expression in storm flux
so it can be used to create/define a component.
For example:
- id: kafkaSpoutFields
className
Hi,
Would like to know if there is a way to write a lambda expression in storm flux
so it can be used to create/define a component.
For example:
- id: kafkaSpoutFields
className: org.apache.storm.tuple.Fields
constructorArgs:
- [ key, value ]
- id
Hi
I have storm and flux running in 2.0.0 version, and in storm it use
DefaultResourceAwareStrategy as schedule policy, I have several questions here:
1. I noticed the deployed topology worker number is less than the
"topology.workers" I configured in topology flux yaml file, an
Hi
I have storm and flux running in 2.0.0 version, and in storm it use
DefaultResourceAwareStrategy as schedule policy, I have several questions here:
1. I noticed the deployed topology worker number is less than the
"topology.workers" I configured in topology flux yaml file, an
Hi, I am creating a topology in flux, using yaml.
I am trying to get the latest offset from Kafka so that storm can process
only the latest messages, for that, I am using static factory methods. Here
is my topology components:
- id: "startingOffsetTime"
className: "kafka.ap
in chooseTasks you
are expected to pick a task id.
From: Rajeev
Reply-To: "user@storm.apache.org"
Date: Wednesday, April 4, 2018 at 7:17 PM
To: "user@storm.apache.org"
Subject: Custom Grouping using flux
I am using a custom grouping that implements CustomStreamGrouping in
) {
long valueToGroup = Long.parseLong( values.get( 1 ).toString() );
long mod = valueToGroup % numberOfTasks;
boltIds.add( ( int ) mod );
}
return boltIds;
}
This is how i use in my flux yaml file under streams:
- name: "SampleStream"
from: "Spout01"
Hi All,
Have a couple of doubts. Can someone please let me know the below:
1. Can we specify the logging level while deploying the topology using flux
?
2. Can we specify the dynamic logging indefinitely for a topology without
specifying the timeout ?
Regards,
Rajeev.
Hi Priyank,
This works great.. thanks..
Regards,
Rajeev.
On Tue, 6 Feb 2018 at 08:37, Priyank Shah wrote:
> Hi Rajeev,
>
>
>
> In that case may be you can add a method called setGroup on your
> spout/bolt and set the value to 4 in flux using configMethods a
That is occurring from shutting down Supervisor, which wouldn't hurt
anything I think. I agree the message looks strange and may feel like
actual error message, so better to reword log message and adjust log level
in this case.
2018년 2월 7일 (수) 오후 12:50, Suman Mallela 님이 작성:
> Hi,
>
>
>
> I have b
Hi Rajeev,
In that case may be you can add a method called setGroup on your spout/bolt and
set the value to 4 in flux using configMethods and then override
getComponentConfiguration method in your spout/bolt to return a map that has
the configuration that you would add in first place using
Hi Priyank,
Thanks for the reply.
I think the configMethods can be used within the spout or bolt codes.. but
i need to use the configuration for my custom scheduler like how we can do
it from java topology builder.
Can we do the same in flux ?
Regards,
Rajeev.
On Tue, 30 Jan 2018 at 03:32
I misread your first question in previous email.
Does not look like setting number of tasks is supported(or straightforward)
using flux yaml. Number of executors is supported
From: Priyank Shah
Reply-To: "user@storm.apache.org"
Date: Monday, February 5, 2018 at 10:56 AM
Hi,
Please find answers inline and check out
http://storm.apache.org/releases/2.0.0-SNAPSHOT/flux.html for more information
From: Rajeev
Reply-To: "user@storm.apache.org"
Date: Monday, February 5, 2018 at 2:15 AM
To: "user@storm.apache.org"
Subject: Number of Tasks i
Hi,
1) Can anyone tell me how to specify the number of tasks for a spout or
bolt in a Flux yaml file ?
2) Does parallelism in flux yaml file mean the number of executor for the
spout or bolt ?
3) Can we use custom scheduler to schedule the topology based on flux yaml ?
Regards,
Rajeev.
: Rajeev
Reply-To: "user@storm.apache.org"
Date: Saturday, January 27, 2018 at 8:36 PM
To: "user@storm.apache.org"
Subject: addConfiguration in flux
Hi,
How can we specify the addConfiguration values that can be used in custom
scheduling under flux ?
For example:
I have a c
Hi,
How can we specify the addConfiguration values that can be used in custom
scheduling under flux ?
For example:
I have a custom scheduler that implements IScheduler.
I have the below under my Topology.java:
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout
Hi,
1. Can we forcibly place specific workers to use only a subset of machines
from the cluster ?
Can this be done using Flux ?
2. Can we place specific components (spouts and bolts) on the machines that
we want ?
Regards,
Rajeev.
Thank you Priyank, appreciate it!
From: Priyank Shah [mailto:ps...@hortonworks.com]
Sent: Friday, November 10, 2017 5:53 PM
To: user@storm.apache.org
Subject: Re: storm flux API - emit to specific streamId
You can provide it under grouping. Check GroupingDef. Pasting an example below.
streams
streamId: “mystream”
From: "Huang, Zhipeng"
Reply-To: "user@storm.apache.org"
Date: Friday, November 10, 2017 at 2:56 PM
To: "user@storm.apache.org"
Subject: storm flux API - emit to specific streamId
Hi Apache Storm Team,
May I have a quick question – how to c
Hi Apache Storm Team,
May I have a quick question - how to configure the streamId in yaml file,
so in Bolt we can emit the tuple to specified stream? Thanks!
Currently the StreamDef don't have "streamId" available in V1.1.1, or even on
Github.
public class StreamDef {
private String name;
See https://github.com/apache/storm/pull/2139/files
On 14 February 2017 at 09:28, anshu shukla wrote:
> +1
>
> On Fri, Aug 26, 2016 at 4:48 PM, Nathan Trevivian
> wrote:
>
>> Hi all,
>>
>> I've been using Flux to configure my topologies, but have noticed
On Windows, trying the following:
mvn compile exec:java -Dexec.args="--local -c 'test.test=test1' --sleep 40
--resource /topology.yaml"
I get:
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java
(default-cli) on project StorageTopology: An exception occured while ex
Probably asked before, but cannot find the answer online -
Can I declare a Spout/Bolt with Flux that emits multiple named streams?
I am using ShellSpout and ShellBolt, any additional gotcha there?
Thanks -
Mauro.
Hello -
I am using FluxShellSpout to call into .net core using the
https://github.com/ziyunhx/storm-net-adapter multilang wrapper -
Question: how do I set the TOPOLOGY_TICK_TUPLE_FREQ_SECS value for a bolt I
configure in Flux via FluxShellSpout ?
I guess I can set the
Working on a Storm project using C# for Bolts and Spouts - as I have a
dependency on .Net/Mono for I/O -
We are using Flux for the topologies and FluxShellSpout to invoke a .net 4.5
executable that uses the storm net adapter
(https://github.com/ziyunhx/storm-net-adapter)
It works when
ontent.com/apache/storm/master/bin/storm-config.cmd
from the master branch
https://issues.apache.org/jira/browse/STORM-2451
bye, Alexander
On 06-Apr-17 23:24, Mauro Giusti wrote:
>
> Hi all –
>
> I am getting the following error when running Flux:
>
>
>
> ERROR StatusLogg
Hi all -
I am getting the following error when running Flux:
ERROR StatusLogger Unable to access file:///log4j2/cluster.xml
java.io.FileNotFoundException: \log4j2\cluster.xml (The system cannot find the
path specified)
at java.io.FileInputStream.open0(Native Method)
at
That seems not possible even using ShellSpout/ShellBolt directly, since
there's no way to pass metrics to register. I tracked the history, and
registering method was there but got removed long time ago.
If we can define the way correctly, Flux might be able to register metrics
too.
Seems li
Hi,
I'm using an older version of storm (0.9.4) and flux so I'm not sure this
will work on newer versions. I also don't know anythign about
FluxShellBolts. That said I put "topology.metrics.consumer.register" in
the config section of flux file.
Hi Marc,
Currently there is not a way to directly register metrics using Flux. It is
something that is probably doable, though. Feel free to file a JIRA to add this
functionality.
-Taylor
> On Feb 15, 2017, at 1:49 PM, Marc Zbyszynski wrote:
>
> Hello Everyone,
>
> I a
Hello Everyone,
I am using flux to deploy multilang topologies with a lot of FluxShellBolt
instances. Do you know if there is any way to register a metric directly in
the flux yaml without writing any Java code? I know that the multilang
protocol supports submitting data to metrics that have been
+1
On Fri, Aug 26, 2016 at 4:48 PM, Nathan Trevivian
wrote:
> Hi all,
>
> I've been using Flux to configure my topologies, but have noticed that it
> doesn't let you specify the number of tasks against executors- only a
> parallelism hint.
>
> When previously co
Sure, I will open a pull request and ticket.
On Feb 2, 2017 9:45 PM, "Jungtaek Lim" wrote:
> Looks great! Thanks for sharing.
>
> I think it's worth to include to the submodule of flux or one of page in
> UI, only If you are open to contribute your work to Storm
AFAIK it's not implemented yet in storm. There's a plan for a swap command
to swap in an updated topology on the fly with zero downtime, but I have no
idea when will that be available.
On Feb 4, 2017 11:22 AM, "anshu shukla" wrote:
> Hello,
>
> I was playing arou
Hello,
I was playing around with Flux for running DAGs with Storm. I am
curious to know that Is there any way to *update the running topology on
the fly using Flux* (something like rebalance to update parallelism).
*I found this in the documentation:-*
*" Flux aims to alleviate that pa
Looks great! Thanks for sharing.
I think it's worth to include to the submodule of flux or one of page in
UI, only If you are open to contribute your work to Storm project.
- Jungtaek Lim (HeartSaVioR)
2017년 2월 3일 (금) 오후 12:03, Xin Wang 님이 작성:
> Hi Ambud,
>
> Thanks for you
Hi Ambud,
Thanks for your nice work. I tested it. Looks good. This can be a useful
tool for flux users.
- Xin
2017-02-03 5:08 GMT+08:00 Ambud Sharma :
> Put together a simple webpage to visualize Flux YAML files to help
> troubleshooting and development of Flux based topologies.
>
Put together a simple webpage to visualize Flux YAML files to help
troubleshooting and development of Flux based topologies.
https://github.com/ambud/flux-viewer
Hi All,
Can any one guide me with JDBC connection to Phoenix, using Flux &
storm-jdbc module of Storm?
Below is YAML code, I am trying to build. But I am facing issue with "
*com.google.common.collect.Maps"*
*+++*
*# Components*
*compon
Hi,
I'm reccently saw Flux in storm 1.0.2 and I wanted to convert my topology.
I've just an issue with a CustomGrouping Object.
Previous, I instanciate it like this :
builder.setBolt("fetcher", new FetcherBolt(), numWorkers)
.customGrouping("spout", new
Hi all,
I've been using Flux to configure my topologies, but have noticed that it
doesn't let you specify the number of tasks against executors- only a
parallelism hint.
When previously configuring topologies using code, you could set the number
of executors, and then specify the numbe
Hi everybody,
I am in the process of migrating to the new KafkaSpout
(https://github.com/apache/storm/tree/master/external/storm-kafka-client) but I
am having trouble setting it up with Flux. While the builder classes like
"org.apache.storm.kafka.spout.KafkaSpoutConfig$Builder" clas
Hello,
Is there a way to use Flux to define topologies on a storm clojure project ?
Maybe someone can share some examples ?
Thanks
.java
On May 17, 2016 4:48 AM, "Joe Darkless" wrote:
> Hello,
>
> I'm trying to find out how to write topology in YAML that Flux proceses
> and which has multiple output streams (not just default).
> I'm using python, so they are FluxShellSpouts.
>
Hello,
I'm trying to find out how to write topology in YAML that Flux proceses
and which has multiple output streams (not just default).
I'm using python, so they are FluxShellSpouts.
Here is a brief scheme:
|[first_spout] [second_spout] \__ __/ | [pr
With regards to the first question, the purpose of flux is to dynamically
create a topology for submission to nimbus. It serves to replace the
topology main() method. It's is suitable for setting up configuration you
use for connecting in prepare(), but it's not suitable for cr
Hi,
after using storm 0.9.4 for some time now, I finally had a look at 0.10.0-beta.
One of the most interesting new features for me was FLUX. The documentation
states that
"...components are roughly analagous to Spring beans"
and in the flux-examples
(https://github.com/apache/
is
> correct.
> >>
> >> If you’d like, feel free to open a JIRA and optionally a pull request
> for a fix. Otherwise, I can take care of it.
> >>
> >> -Taylor
> >>
> >> On Jun 17, 2015, at 4:07 PM, Romeo Nocon wrote:
> >>
>
porting that. It’s a bug, and your approach for a fix is
>> correct.
>>
>> If you’d like, feel free to open a JIRA and optionally a pull request for a
>> fix. Otherwise, I can take care of it.
>>
>> -Taylor
>>
>> On Jun 17, 2015, at 4:07 PM,
wrote:
>
>> Hi,
>>
>> I'm testing migrating over a topology I have to flux. The
>>
>> spout:
>> - id: "spout"
>>
>> bolts:
>> - id: "bolt_A"
>> className: "com.blah.boltA"
>> parallelism: 1
#x27;m testing migrating over a topology I have to flux. The
>
> spout:
> - id: "spout"
>
> bolts:
> - id: "bolt_A"
>className: "com.blah.boltA"
>parallelism: 1
> - id: "bolt_B"
>className: "com.blah.boltB"
>
Hi,
I'm testing migrating over a topology I have to flux. The
spout:
- id: "spout"
bolts:
- id: "bolt_A"
className: "com.blah.boltA"
parallelism: 1
- id: "bolt_B"
className: "com.blah.boltB"
parallelism: 1
- id: &
Hi All,
Two questions:
1) what version of hadoop does flux work with? My assumption was that it
should not matter, but I am getting the following error when I use a
hdfsbolt with flux which I assume implies some version mismatch. I don't
have this issue if build a topology straightforward
er about the session
> termination. I must be missing something on my side. I'll look around to see
> if I can find anything.
>
> thank you
> Clay
>
>
>
>> On Tue, May 26, 2015 at 11:53 AM, P. Taylor Goetz wrote:
>> I just tested with the following command (from the
the following command (from the “flux-examples”
> directory):
>
> storm jar target/flux-examples-0.3.1-SNAPSHOT.jar
> org.apache.storm.flux.Flux ./src/main/resources/simple_wordcount.yaml
>
> and it ran fine, logging the running count totals at the INFO level:
&g
I just tested with the following command (from the “flux-examples” directory):
storm jar target/flux-examples-0.3.1-SNAPSHOT.jar org.apache.storm.flux.Flux
./src/main/resources/simple_wordcount.yaml
and it ran fine, logging the running count totals at the INFO level:
95957 [Thread-11-bolt-2
for local topologies with the
> `-s` option followed by the sleep time in milliseconds? If so, with what
> value? By default, flux will run local mode topologies for 60 seconds.
>
> 2) It looks like you are trying to use KafkaConfig with constructor
> arguments for SpoutConfig. The S
Hi Clay,
1) Are you overriding the default sleep time for local topologies with the `-s`
option followed by the sleep time in milliseconds? If so, with what value? By
default, flux will run local mode topologies for 60 seconds.
2) It looks like you are trying to use KafkaConfig with
Hi All,
I am trying to test flux module for writing template driven topologies. I
setting the topologies in localcluster mode.
1) using template simple_wordcount.yaml
2) using kafka_spout.yaml (using TestBolt)
With (1) I don't get any output form TestBolt and the topology exits.
With (2),
67 matches
Mail list logo
