On Tue, Aug 14, 2018 at 12:09 PM Jan Grashöfer wrote:
>
> On 13/08/18 18:24, Jon Siwek wrote:
> > Old Worker:
> >
> >Cluster::relay_rr(Cluster::proxy_pool, my_event);
> >
> > New Worker:
> >
> >Broker::publish(Cluster::rr_topic(Cluster::proxy_pool), my_event);
>
> That doesn't look like an
On 13/08/18 18:24, Jon Siwek wrote:
> Old Worker:
>
>Cluster::relay_rr(Cluster::proxy_pool, my_event);
>
> New Worker:
>
>Broker::publish(Cluster::rr_topic(Cluster::proxy_pool), my_event);
That doesn't look like an API simplification to me ;D
> Even Newer Worker:
>
>Broker::publis
On Tue, Aug 14, 2018 at 10:51 -0500, Jonathan Siwek wrote:
> Not sure, is Broker::auto_publish() currently able to do the same thing?
Hmm .. Good point. Scope is different between the two (event vs topic)
but the effect is similar in the end.
> I can also see the opposite being intuitive: If
On Tue, Aug 14, 2018 at 10:13 AM Robin Sommer wrote:
> One more question: what about raising published events locally as well
> if the sending node is subscribed to the topic? I'm kind of torn on
> that. I don't think we want that as a default, but perhaps as an
> option, either with the publish(
On Mon, Aug 13, 2018 at 13:55 -0500, Jonathan Siwek wrote:
> associating node IDs with subscription state and also message state
> (push node IDs into messages upon receipt before forwarding),
Yeah, that sounds like the right direction. Some reading might be
worthwile doing here, there are qui
On Mon, Aug 13, 2018 at 12:56 PM Azoff, Justin S wrote:
> > Broker::publish(Cluster::worker_topic, my_event);
> The one tricky case is doing that on the manager. While the manager is fully
> connected to all workers,
> you really want to offload the fanning out of messages to one of the proxi
> On Aug 13, 2018, at 12:24 PM, Jon Siwek wrote:
>
> Even Newer Worker:
>
> Broker::publish(Cluster::worker_topic, my_event);
>
> See any problems there?
That's nice and simple :-)
Assuming that can send the events around in the most efficient way possible,
that's perfect.
The one tricky
On Mon, Aug 13, 2018 at 8:09 AM Jan Grashöfer wrote:
>
> On 10/08/18 17:12, Robin Sommer wrote:
> > I hear you, but I think I haven't quite understood the concern yet.
> > Can you give me an example where the difference matters? What's
> > different between publishing intel events to bro/cluster/w
On Fri, Aug 10, 2018 at 11:47 AM Azoff, Justin S wrote:
> If relay is removed how does a script writer efficiently get an event from
> one worker (or manager)
> to all of the other workers?
Old Worker:
Cluster::relay_rr(Cluster::proxy_pool, my_event);
New Worker:
Broker::publish(Cluster:
On 10/08/18 17:12, Robin Sommer wrote:
> I hear you, but I think I haven't quite understood the concern yet.
> Can you give me an example where the difference matters? What's
> different between publishing intel events to bro/cluster/worker/intel
> vs bro/cluster/worker if both go to all workers? O
> On Aug 10, 2018, at 11:55 AM, Robin Sommer wrote:
>
>
> Ok, let's make that change then, I think removing relay() will help
> for sure making the API easier.
If relay is removed how does a script writer efficiently get an event from one
worker (or manager)
to all of the other workers?
—
J
On Fri, Aug 10, 2018 at 10:24 -0500, Jonathan Siwek wrote:
> Or is it a matter of "if a user needed it for something, then it's
> available" ?
Yeah, including matching expectations: if there's a
"bro/cluster/worker" topic, I'd expect I can publish there to reach
all the workers (from anywhere)
On Fri, Aug 10, 2018 at 8:29 AM Jan Grashöfer wrote:
> > Let me try to phrase it differently: If there's already a topic for a
> > use case, it's better to use it. That's easier and less error-prone.
> > So if, e.g., I want to send my script's data to all workers,
> > publishing to bro/cluster/wo
On Thu, Aug 9, 2018 at 1:29 PM Robin Sommer wrote:
> > (1) enable the "explicit/manual" forwarding by default?
>
> Coming from that assumption above, I'd say yes here, doing it like you
> suggest: differentiate between forwarding and locally raising an event
> by topic. Maybe instead of adding it
On Fri, Aug 10, 2018 at 15:22 +0200, Jan Grashöfer wrote:
> different purposes. If that is still a design goal, it feels like the
> structure of a cluster could be more volatile than it used to be.
It is, and we have some of that, and I think it fits in with the
discussion here too. In my mind
On 08/08/18 17:48, Robin Sommer wrote:> I think it's safe to assume we
have the cluster structure under our
> own control; it's whatever we configure it to be. That's something
> that's easier to change later than the API itself. Said differently:
> we can always adjust the connections and topics
Yeah, and let me add one thing: What if as a starting point for
modeling things, we assumed that we have global topic-based routing
available. Meaning if node A publishes to topic X, the message will
show up at all nodes that are subscribed to topic X anywhere, no
matter what the topology --- Broke
On Wed, Aug 8, 2018 at 2:50 PM Robin Sommer wrote:
> > * enable message forwarding by default (meaning re-implement the one
> > or two subscription patterns that might create a cycle)
>
> Haven't quite made up my mind on this one. In principlel yes, but
> right now a host needs to be subscribed t
On Wed, Aug 08, 2018 at 12:36 -0500, Jonathan Siwek wrote:
> * publish() API simplifications/compressions (pending decision on
> exactly what those should be)
Yeah, with an eye on the semantics for forwarding (now and later),
and whether to raise published events locally as well if the host is
On Wed, Aug 8, 2018 at 10:55 AM Robin Sommer wrote:
> That's actually something I realized yesterday: we don't have direct
> worker-to-worker communication right now, correct? A worker cannot
> just publish to "bro/cluster/workers".
Right, here's a crude graphic of the cluster layout from the do
On Wed, Aug 8, 2018 at 10:53 AM Robin Sommer wrote:
>
> Yeah, I realize that. A direct port of the old logic was of course the
> goal so far, with the drawbacks of that approach accepted &
> understood. That's what's in place now; that's great and exactly as
> planned. We can get 2.6 out this way,
On Wed, Aug 08, 2018 at 14:20 +, Justin Azoff wrote:
> There's also a bunch of places that I think were written standalone first and
> then updated to work on a cluster in
> place resulting in some awkwardness..
Yeah, indeed, that's another other source of complexity with these
scripts.
Yeah, I realize that. A direct port of the old logic was of course the
goal so far, with the drawbacks of that approach accepted &
understood. That's what's in place now; that's great and exactly as
planned. We can get 2.6 out this way, and it'll be fine.
My point is that now also seems like a goo
On Tue, Aug 07, 2018 at 12:05 +0200, Jan Grashöfer wrote:
> What I can recall, it's about simplifying the API in the light of
> multi-hop routing, which is not fully functional yet.
To level up a bit, what I'm hoping for is that we can find some easy
ways to simplify the API a bit more now, wit
> On Aug 6, 2018, at 3:50 PM, Robin Sommer wrote:
>
>- Relaying is hardly used.
>
>
>- There's a lot of checks in publishing code of the type "if I am
> (not) of node type X".
I think these 2 are somewhat related. Since there weren't higher level things
like relaying, in order
On Mon, Aug 6, 2018 at 3:00 PM Robin Sommer wrote:
> Overall I have to say I found it pretty hard to follow this all
> because we don't have much consistency right now in how scripts
> structure their communication. That's not surprising, given that we're
> just starting to use all this, but it s
On Mon, Aug 6, 2018 at 1:57 PM Robin Sommer wrote:
> I have another question about this specific case: we use relay_rr()
> only for sending Intel::insert_indicator. Intel::remove_indicator gets
> published normally through auto_publish(). Why the difference?
Potentially no reason other than no o
To be honest, I have somehow lost track of the discussion. What I can
recall, it's about simplifying the API in the light of multi-hop
routing, which is not fully functional yet.
Regarding multi-hop routing I am even not sure what the actual goal is
that we are currently aiming at. However, fro
On Mon, Jul 30, 2018 at 09:01 -0700, I wrote:
> Is there a summary somewhere of what events & topics the cluster nodes
> are currently exchanging?
So I went through the exercise of collecting this information: what
connections do we have between nodes, who's subscribing to what, and
who's publi
On Fri, Aug 03, 2018 at 15:57 -0500, Jonathan Siwek wrote:
> Another use is hidden within Cluster::relay_rr():
Yeah, though at least from an API perspective this is different: The
caller gives relay_rr() only one topic to send to (indicator_topic).
It's then using a different topic internally
On Fri, Aug 3, 2018 at 12:22 PM Robin Sommer wrote:
> On Fri, Jul 27, 2018 at 10:39 -0700, I wrote:
>
> > Broker::relay(change_topic, change_topic, Config::cluster_set_option, ID,
> > val, location);
>
> Can somebody remind me what the use-case is for changing the topic on
> relay? Grepping over
On Fri, Jul 27, 2018 at 10:39 -0700, I wrote:
> Broker::relay(change_topic, change_topic, Config::cluster_set_option, ID,
> val, location);
Can somebody remind me what the use-case is for changing the topic on
relay? Grepping over our standard scripts, I see only one use of
relay(), and that'
On Mon, Jul 30, 2018 at 13:30 -0500, Jonathan Siwek wrote:
> Seems clunky and could get dicey
Agreed. :) It'd just be a heuristic to catch some obvious errors. I
don't think there's more we can do, we can't really catch loops
statically by looking at the code.
Robin
--
Robin Sommer * Coreli
On Mon, Jul 30, 2018 at 12:16 PM Robin Sommer wrote:
> > I'd be more comfortable if one could automate answering the question:
> > "if I add a subscription to a given node in the network, will I create
> > a cycle?".
>
> Hmm ... What about a test mode where we'd spin up a dummy cluster
> (similar
On Mon, Jul 30, 2018 at 11:15 -0500, Jonathan Siwek wrote:
> I don't see why not, but it takes planning and prudence on everyone's
> part (including users) to not break that rule.
Yeah, question is we can pre-configure the cluster so that user's
don't need to worry about it most of the time.
On Mon, Jul 30, 2018 at 11:02 AM Robin Sommer wrote:
> True, although it's not cycles in the connection topology that matter,
> it's cycles in topic subscriptions.
Right, good point.
> I need to think about this a bit
> more (and I need to remind myself how our topics currently look like)
I th
On Fri, Jul 27, 2018 at 7:30 PM Azoff, Justin S wrote:
>
>
> > On Jul 27, 2018, at 6:10 PM, Jon Siwek wrote:
> >
> > On Fri, Jul 27, 2018 at 3:55 PM Azoff, Justin S wrote:
> >
> >> I do agree that there's room for a lot of simplification, for example a
> >> worker broadcasting a message efficie
On Fri, Jul 27, 2018 at 14:47 -0500, Jonathan Siwek wrote:
> Broker does not yet have automatic multihop where subscriptions are
> globally flooded automatically.
Yep, that's what I meant: dynamic multihop where each node tracks what
its peers are subscribing to, and forwards messages independent
> On Jul 27, 2018, at 6:10 PM, Jon Siwek wrote:
>
> On Fri, Jul 27, 2018 at 3:55 PM Azoff, Justin S wrote:
>
>> I do agree that there's room for a lot of simplification, for example a
>> worker broadcasting a message efficiently to all
>> other workers needs to do something like this from the
On Fri, Jul 27, 2018 at 3:55 PM Azoff, Justin S wrote:
> I do agree that there's room for a lot of simplification, for example a
> worker broadcasting a message efficiently to all
> other workers needs to do something like this from the docs:
>
> Cluster::relay_rr(Cluster::proxy_pool, "e
> On Jul 27, 2018, at 1:39 PM, Robin Sommer wrote:
>
> I'm wondering if we should give it another try to simply this API
> while we still can (i.e., before 2.6 goes out). To me, the most
> intuitive publish operation is "send to topic T and propagate to
> everybody subscribed to that topic". I'd
On Fri, Jul 27, 2018 at 12:40 PM Robin Sommer wrote:
> I'm wondering if we should give it another try to simply this API
> while we still can (i.e., before 2.6 goes out). To me, the most
> intuitive publish operation is "send to topic T and propagate to
> everybody subscribed to that topic". I'd
42 matches
Mail list logo
