Re: Should Flume integration be behind a profile?

>> >> a profile. Like with Kafka 0.8. (This is why I'm raising it again to >> the >> >> whole list.) >> >> >> >> Any preferences among: >> >> 1. Put Flume behind a profile, remove examples, deprecate >> >> 2. Put Fl

Re: Flume integration

Hi Mich, Thanks. I would prefer not to add another system into the mix as we currently don't use kafka at all. We are still in the prototype phase at the moment and it seems to be working well though it doesn't like you restrating the flume sink part without restarting the SPARK application.

Re: Flume integration

Hi Ian, Flume is great for ingesting data into HDFS and Hbase. However, that is part of batch layer. For real time processing, I would go through Kafka into spark streaming. Except your case, I have not established if anyone else does Flume directly into Spark? If so how mature is it. Thanks

Re: Flume integration

*-Ian* Hi While I am following this discussion with interest, I am trying to comprehend any architectural benefit of a spark sink. Is there any feature in flume makes it more suitable to ingest stream data than sppark streaming, so that we should chain them? For example does it help

Re: Flume integration

Hi While I am following this discussion with interest, I am trying to comprehend any architectural benefit of a spark sink. Is there any feature in flume makes it more suitable to ingest stream data than sppark streaming, so that we should chain them? For example does it help durability or

Re: Flume integration

Thanks Ian. Was your source of Flume IBM/MQ by any chance? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Flume integration

Hi Mich, Yes, i managed to resolve this one. The issue was because the way described in the docs doesn't work properly as in order for the Flume part to be notified you need to set the storageLevel on the PollingStream like JavaReceiverInputDStream flumeStream =

Re: Flume integration

Hi Ian, Has this been resolved? How about data to Flume and then Kafka and Kafka streaming into Spark? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Flume integration

Hi, I'm currently trying to implement a prototype Spark application that gets data from Flume and processes it. I'm using the pull based method mentioned in https://spark.apache.org/docs/1.6.1/streaming-flume-integration.html The is initially working fine for getting data from Flume, however

Re: query on Spark + Flume integration using push model

are in JSON format, but the Spark + Flume integration [1] document only refer to Avro sink. [1] https://spark.apache.org/docs/latest/streaming-flume-integration.html I looked at some of the examples online, and they all refer to avro type: agent.sinks.avroSink.type = avro If I set the type

Re: query on Spark + Flume integration using push model

up the data. My events are in JSON format, but the Spark + Flume integration [1] document only refer to Avro sink. [1] https://spark.apache.org/docs/latest/streaming-flume-integration.html I looked at some of the examples online, and they all refer to avro type: agent.sinks.avroSink.type

query on Spark + Flume integration using push model

Hello all, I'm trying to configure the flume to push data into a sink so that my stream job could pick up the data. My events are in JSON format, but the Spark + Flume integration [1] document only refer to Avro sink. [1] https://spark.apache.org/docs/latest/streaming-flume-integration.html I

Spark and Flume integration - do I understand this correctly?

://apache-spark-user-list.1001560.n3.nabble.com/Spark-and-Flume-integration-do-I-understand-this-correctly-tp10879.html Sent from the Apache Spark User List mailing list archive at Nabble.com.