Flume Community,
I created a Flume Plugin with multiple components that complements the
current version of Apache Flume.
This was necessary as part of a personal project as I working on.
It is code named - Flume Jambalaya
Jambalaya is a standalone Apache Flume plugin that contains a variety of
Congratulations Roshan, well deserved.
*Author and Instructor for the Upcoming Book and Lecture Series*
*Massive Log Data Aggregation, Processing, Searching and Visualization with
Open Source Software*
*http://massivelogdata.com*
On Wed, Sep 25, 2013 at 4:54 PM, Roshan Naik
Anat,
Once the event is written to the Sink you cannot intercept it any longer
unless that sink is marking the beginning of another source in a
multi-agent flow.
In the Flume architecture, the interceptors sit between the Source and the
Channel, so their role is to modify or eliminate any event
Hello Zoraida,
What sources are you events coming from?
I have a feeling they are coming from SpoolingDirectory and the events
contains newline characters (even delimiter).
If this is the case, you are going to see the events split up whenever the
parser encounters the delimiter.
*Author and
Andrei,
Check out the monitoring tool here and see if it helps
http://flume.apache.org/FlumeUserGuide.html#monitoring
*Author and Instructor for the Upcoming Book and Lecture Series*
*Massive Log Data Aggregation, Processing, Searching and Visualization with
Open Source Software*
Zoraida,
You can take a look at the source code for the following interceptors to
see how they are implemented.
http://flume.apache.org/releases/content/1.4.0/apidocs/org/apache/flume/interceptor/package-summary.html
They do have member fields (attributes) and they are mostly private.
will be very appreciated.
De: Israel Ekpo isr...@aicer.org
Responder a: Flume User List user@flume.apache.org
Fecha: martes, 27 de agosto de 2013 16:29
Para: Flume User List user@flume.apache.org
Asunto: Re: Events being cut by flume
Hello Zoraida,
What sources are you events coming from
Deepak,
The sample configuration you have looks valid
Try it out and let us know if you are experiencing any problems
More information on how to configure the Exec Source is available here.
http://flume.apache.org/FlumeUserGuide.html#exec-source
*Author and Instructor for the Upcoming Book
Sunita,
Depending on your level of comfort, you can do one of the following:
1. Use Python to fetch your data and then send the events via HTTP to the
Flume HTTP Source [1]
2. Use Java to create a custom source [6] in Flume that handles the data
fetching and then puts it in a channel [3] so that
Otis,
I have not implemented this yet but check out the Apache Commons IO Library.
It has a set of classes that supports tailing files that could accomplish
what you are trying to do.
http://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/input/Tailer.html
How about putting Flume behind a reverse proxy that uses HTTPS using Nginx
Ashish paliwalash...@gmail.com wrote:
So am I :) I am just into my 2nd week of reading about Flume. If I were in
your place, I would have added Jetty SSL example code to HTTP source and make
it work. Other approaches
been sended to the agent_2nd).
3.stop the agent_2nd.
4.delete the log.
5.start the agent_2nd.
6.send the log from agent_1st 2 agent_2nd.(There is nothing in the log
folder. And the flume has't error info).
Thanks a lot!
2013-05-06
--
sam
--
*Israel Ekpo
You should send it to user-unsubscr...@flume.apache.org instead.
On Mon, May 6, 2013 at 2:51 AM, Jeff Lord jeffrey.l...@gmail.com wrote:
If you are just starting out, I think you should grab the latest version of
Flume (1.3.1)
http://flume.apache.org/download.html
Version 1.2.0 is an older version so working with the latest version is
better.
Try it out and send your feedback.
On 24 April 2013 10:15, Shekhar Sharma
...Will update if
i find something..
Regards,
Som Shekhar Sharma
+91-8197243810
On Wed, Apr 24, 2013 at 9:04 PM, Israel Ekpo isr...@aicer.org wrote:
If you are just starting out, I think you should grab the latest version of
Flume (1.3.1)
http://flume.apache.org/download.html
Version
so that the
correct version is picked up and used in your application.
On 24 April 2013 12:52, Israel Ekpo isr...@aicer.org wrote:
Which version of Flume are you using?
If this is something that just changed recently, consider moving the new
jar to the end of your class path so that it does
Hello Venkat,
Your question is more appropriate for the users mailing list so I have
changed the list in this reply.
Going forward, you can use the following as a guide when sending emails to
the lists:
For questions about how to use or configure Apache FLUME or if you are
experiencing issues
David,
In addition to what has already been said, if you take a look at your flume
log files, you should be able to see exception messages that explain why
this is happening.
On 22 April 2013 11:11, David Quigley dquigle...@gmail.com wrote:
Hi,
I am using flume to write events from
is for the HTTP Source
which I've already been able to verify as working.
Thanks,
Paul
--
*From:* Israel Ekpo [mailto:isr...@aicer.org]
*Sent:* Friday, April 12, 2013 1:15 PM
*To:* user@flume.apache.org
*Subject:* Re: spoolDir source problem
Paul,
I have the following
Paul,
Phil Scala had a similar issue today on his Windows machine.
He has created a JIRA issue and has submitted a patch that will be checked
in soon.
https://issues.apache.org/jira/browse/FLUME-1993
You can compare this with your changes to see if it helps.
Paul,
If Flume 1.3.1 is what you are looking for, you don't have to build it from
source.
You can just download it directly from the site. Its already released.
Paul Chavez pcha...@verticalsearchworks.com wrote:
Hello,
I've been using flume on Windows to send logs to an avro source for
The specific version used by Flume 1.3.1 is version 1.0.1
You can grab the jar file from this link below and drop it in
the $FLUME_HOME/lib directory where all the other jar files are located.
http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/1.0.1/hadoop-core-1.0.1.jar
On 10 April
and there are no startup errors now.
Thanks to you, Brock and Hari.
-Paul
--
*From:* Israel Ekpo [mailto:isr...@aicer.org]
*Sent:* Wednesday, April 10, 2013 12:07 PM
*To:* user@flume.apache.org
*Subject:* Re: FileChannel on Windows
Paul,
If Flume 1.3.1 is what you are looking
Hello Everyone,
I have been giving this a great deal of thought over the last 3 weeks.
I would really appreciate feedback on how we should proceed on this from
users, developers (committers), and project management perspectives.
It would really be nice for the documentation for the project to
Sandeep,
Excellent questions.
You asked what problem Flume is trying to solve?.
It think the more appropriate question is what problem you are trying to
solve?
This will go a long way in helping us understand which components of Flume
you may need and how you need to set it up.
Are you using
To add to Hari's recommendation, if the beginning of each log event is
consistent in terms of timestamps and log levels etc (-MM-dd HH:ii:ss
DEBUG or INFO or WARNING or FATAL) you can create a regex that scans for
these string patterns and use that as delimiters instead of line endings
On Mar
Nguyen,
It might be helpful, if the original log data is saved in a separate
directory first and then you can use a separate script/program to send the
*diffs* periodically to the directory being spooled by Flume.
If the log files are rolled, then your script might need to be aware of the
27 matches
Mail list logo
