Hey Joe, I may try 1.7 if you think it's pretty stable in master. Flume as a relay actually solves our problem pretty well, in that if the Kafka DC becomes unavailable for any reason, the Flume file channel will queue up the entries reliably and replay when the link comes back.
-J Sent via iPhone > On Jun 10, 2016, at 21:25, Joe Lawson <[email protected]> > wrote: > > So 1.7 would support Kafka 0.9. The issues are pretty minimal around the sink > in master. Have you tried it? As for other means flume feels the purest in > not manipulating the data. If you are shipping logs, Logstash (probably a bad > word here :) is an option as it supports Kafka 0.9. > > You could do something more hacky like piping into the Kafka console producer > too. > > I know (elastic's) beats supports Kafka 0.9 in master but don't think it is > released either. Remember Kafka 0.9 came out in Oct or Dec so it's pretty > durn new compared to 0.8 which had over a year to get momentum. Let's not > even talk about 0.10 ;) > > maybe this helps? > > -Joe > > On Jun 10, 2016 10:34 PM, "Jason J. W. Williams" <[email protected]> > wrote: > Yeah. > > Flume (relay)----WAN (Avro SSL)---->Flume (concentrator)--->Kafka. > > The remote Flume is talking to a "concentrator" flume in the Kafka > datacenter, so that the data is transferred via SSL over the WAN, since the > Kafka sink doesn't support SSL in 1.6.0. Otherwise, we could push from the > remote Flume directly into Kafka over the WAN. > > -J > > >> On Fri, Jun 10, 2016 at 8:29 PM, Joe Lawson >> <[email protected]> wrote: >> Do you mean that you are replicating topics over WAN using Flume as a >> shipping agent? Sorry if I misunderstood the scenario. >> >>> On Jun 10, 2016 9:58 PM, "Jason J. W. Williams" <[email protected]> >>> wrote: >>> Thanks Joe that helps. What do you see folks using instead of Flume? We're >>> using largely a WAN relay for Kafka for remote sites that don't warrant a >>> local cluster. >>> >>> -J >>> >>>> On Fri, Jun 10, 2016 at 6:17 PM, Joe Lawson >>>> <[email protected]> wrote: >>>> Flume seems to have lost some stream. It's improving but it feels like the >>>> priorities have become somewhat introspective so I'd say build from master >>>> is you want the latest. >>>> >>>> Don't hold your breath for 1.7. just an opinion. No bearing on reality. >>>> >>>> -Joe >>>> >>>>> On Jun 10, 2016 12:45 PM, "Jason J. W. Williams" >>>>> <[email protected]> wrote: >>>>> Is there an estimated release date for Flume 1.7.0? Having TLS support in >>>>> the Kafka sink would allow us to simplify our topology a lot.=20 >>>>> >>>>> -J >
