Re: Review Request: FLUME-1383: Improve various log messages in FileChannel and HDFSSink

2012-07-18 Thread Jarek Cecho 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6042/#review9278
---

Ship it!


Looks good to me.

Jarcec

- Jarek Cecho


On July 19, 2012, 6:22 a.m., Mike Percy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6042/
> ---
> 
> (Updated July 19, 2012, 6:22 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> ---
> 
> Added file channel name to some log messages & exceptions. Also added timeout 
> length in HDFS Sink. These changes help with debugging in certain cases.
> 
> 
> This addresses bug FLUME-1383.
> https://issues.apache.org/jira/browse/FLUME-1383
> 
> 
> Diffs
> -
> 
>   
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
>  77417ea 
>   
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
>  d65d3da 
>   
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
>  0a94e04 
>   
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
>  21f42be 
>   
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java
>  e81897e 
>   
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
>  024a4a8 
> 
> Diff: https://reviews.apache.org/r/6042/diff/
> 
> 
> Testing
> ---
> 
> Unit tests pass.
> 
> 
> Thanks,
> 
> Mike Percy
> 
>

Re: Review Request: FLUME-1358: Add a regex-based filtering interceptor

2012-07-18 Thread Jarek Cecho 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5898/#review9277
---


Hi Patrick,
thank you very much for your time and effort. Code and tests seems good to me, 
I just have few notes to UserGuide changes:


flume-ng-doc/sphinx/FlumeUserGuide.rst


Tilda characters ("~") are not below entire headline which results in 
compilation error:

System Message: WARNING/2 
(/home/jarcec/projects/apache/flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst,
 line 1737)

Title underline too short.



flume-ng-doc/sphinx/FlumeUserGuide.rst


Could you put "type" property in bold like **type**.


Jarcec

- Jarek Cecho


On July 19, 2012, 2:54 a.m., Patrick Wendell wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5898/
> ---
> 
> (Updated July 19, 2012, 2:54 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> ---
> 
> Adding a regex-based filtering interceptor
> 
> 
> This addresses bug FLUME-1358.
> https://issues.apache.org/jira/browse/FLUME-1358
> 
> 
> Diffs
> -
> 
>   
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java 
> 0c112d0 
>   
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexFilteringInterceptor.java
>  PRE-CREATION 
>   
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexFilteringInterceptor.java
>  PRE-CREATION 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 7967d7c 
> 
> Diff: https://reviews.apache.org/r/5898/diff/
> 
> 
> Testing
> ---
> 
> Unit tests are included
> 
> 
> Thanks,
> 
> Patrick Wendell
> 
>

[jira] [Updated] (FLUME-1383) Improve various log messages in FileChannel and HDFSSink

2012-07-18 Thread Mike Percy (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Percy updated FLUME-1383:
--

Attachment: FLUME-1383-1.patch

> Improve various log messages in FileChannel and HDFSSink
> 
>
> Key: FLUME-1383
> URL: https://issues.apache.org/jira/browse/FLUME-1383
> Project: Flume
>  Issue Type: Improvement
>Affects Versions: v1.2.0
>Reporter: Mike Percy
>Assignee: Mike Percy
>Priority: Minor
> Fix For: v1.3.0
>
> Attachments: FLUME-1383-1.patch
>
>
> Catch-all JIRA for a few patches I have to improve logging and ease 
> troubleshooting in certain cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request: FLUME-1383: Improve various log messages in FileChannel and HDFSSink

2012-07-18 Thread Mike Percy 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6042/
---

Review request for Flume.


Description
---

Added file channel name to some log messages & exceptions. Also added timeout 
length in HDFS Sink. These changes help with debugging in certain cases.


This addresses bug FLUME-1383.
https://issues.apache.org/jira/browse/FLUME-1383


Diffs
-

  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
 77417ea 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
 d65d3da 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
 0a94e04 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
 21f42be 
  
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java
 e81897e 
  
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
 024a4a8 

Diff: https://reviews.apache.org/r/6042/diff/


Testing
---

Unit tests pass.


Thanks,

Mike Percy

Re: Review Request: FLUME-1360: Provide documentation for static interceptor

2012-07-18 Thread Jarek Cecho 


> On July 19, 2012, 5:51 a.m., Hari Shreedharan wrote:
> > +1. Thanks for the patch, Jarcec! Please attach the patch to the jira.

Hi Hari,
thank you very much for your review, I've attached last patch version to the 
jira.

Jarcec


- Jarek


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5879/#review9275
---


On July 16, 2012, 4:53 p.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5879/
> ---
> 
> (Updated July 16, 2012, 4:53 p.m.)
> 
> 
> Review request for Flume, Arvind Prabhakar, Juhani Connolly, Mike Percy, and 
> Hari Shreedharan.
> 
> 
> Description
> ---
> 
> I've documented static interceptor and I also put some examples to other 
> interceptors as well.
> 
> 
> This addresses bug FLUME-1360.
> https://issues.apache.org/jira/browse/FLUME-1360
> 
> 
> Diffs
> -
> 
>   /trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst 1362119 
> 
> Diff: https://reviews.apache.org/r/5879/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

[jira] [Updated] (FLUME-1360) Provide documentation for static interceptor

2012-07-18 Thread Jarek Jarcec Cecho (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated FLUME-1360:
--

Attachment: FLUME-1360.patch

> Provide documentation for static interceptor
> 
>
> Key: FLUME-1360
> URL: https://issues.apache.org/jira/browse/FLUME-1360
> Project: Flume
>  Issue Type: Improvement
>  Components: Docs
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
>Priority: Minor
> Attachments: FLUME-1360.patch, FLUME-1360.patch
>
>
> We've added new static interceptor in FLUME-1276. It would be great to 
> provide it's documentation into user guide.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: FLUME-1360: Provide documentation for static interceptor

2012-07-18 Thread Hari Shreedharan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5879/#review9275
---

Ship it!


+1. Thanks for the patch, Jarcec! Please attach the patch to the jira.

- Hari Shreedharan


On July 16, 2012, 4:53 p.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5879/
> ---
> 
> (Updated July 16, 2012, 4:53 p.m.)
> 
> 
> Review request for Flume, Arvind Prabhakar, Juhani Connolly, Mike Percy, and 
> Hari Shreedharan.
> 
> 
> Description
> ---
> 
> I've documented static interceptor and I also put some examples to other 
> interceptors as well.
> 
> 
> This addresses bug FLUME-1360.
> https://issues.apache.org/jira/browse/FLUME-1360
> 
> 
> Diffs
> -
> 
>   /trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst 1362119 
> 
> Diff: https://reviews.apache.org/r/5879/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

[jira] [Created] (FLUME-1383) Improve various log messages in FileChannel and HDFSSink

2012-07-18 Thread Mike Percy (JIRA) 
Mike Percy created FLUME-1383:
-

 Summary: Improve various log messages in FileChannel and HDFSSink
 Key: FLUME-1383
 URL: https://issues.apache.org/jira/browse/FLUME-1383
 Project: Flume
  Issue Type: Improvement
Affects Versions: v1.2.0
Reporter: Mike Percy
Assignee: Mike Percy
Priority: Minor
 Fix For: v1.3.0


Catch-all JIRA for a few patches I have to improve logging and ease 
troubleshooting in certain cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1379) avro source , hdfs sink, no event writed to hdfs

2012-07-18 Thread Denny Ye (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418085#comment-13418085
 ] 

Denny Ye commented on FLUME-1379:
-

In my opinion, you should monitor the thread status with 'jstack' to confirm 
all the threads have valid activity. 

>  avro source , hdfs sink, no event writed to hdfs
> -
>
> Key: FLUME-1379
> URL: https://issues.apache.org/jira/browse/FLUME-1379
> Project: Flume
>  Issue Type: Question
>  Components: Sinks+Sources
>Affects Versions: v1.1.0, v1.3.0
> Environment: rhel5.7 ,jdk1.6,yum installed flume-ng 
> 1.1.0+120-1.cdh4.0.0  ,mvn install 1.3.
>Reporter: jack li
>  Labels: flume
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> hello everyone!
> 1, source's type is exec  and sink's type is hdfs .the event can write to the 
> hdfs.But when I change the source's type to avro ,no event write to hdfs.
> 2, flume.conf 
> #List sources, sinks and channels in the agent
> weblog-agent.sources = s1
> weblog-agent.sinks = avro-forward-sink01
> weblog-agent.channels = jdbc-channel01
> #define the flow
> #webblog-agent sources config
> weblog-agent.sources.s1.channels = jdbc-channel01
> weblog-agent.sources.s1.type = avro
> weblog-agent.sources.s1.port = 41414
> weblog-agent.sources.s1.bind = 0.0.0.0
> #avro sink properties
> weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
> weblog-agent.sinks.avro-forward-sink01.type = hdfs
> weblog-agent.sinks.avro-forward-sink01.hdfs.path = 
> hdfs://hadoop-master.:9000/user/flume/webtest
> #weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_%p
> #weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_web001
> weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = 
> fl...@hadoop.com
> weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab = 
> /var/run/flume-ng/flume.keytab
> weblog-agent.sinks.avro-forward-sink01.hdfs.maxOpenFiles = 65535
> weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0
> weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0
> weblog-agent.sinks.avro-forward-sink01.hdfs.batchSize = 3
> weblog-agent.sinks.avro-forward-sink01.hdfs.txnEventMax = 2
> weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream
> #channels config
> weblog-agent.channels.jdbc-channel01.type = memory
> weblog-agent.channels.jdbc-channel01.capacity = 20
> weblog-agent.channels.jdbc-channel01.transactionCapacity = 2
> 3,flume.log
> 2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting 
> lifecycle supervisor 1
> 2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting - 
> weblog-agent
> 2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node 
> manager starting
> 2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting 
> lifecycle supervisor 10
> 2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider: 
> Configuration provider starting
> 2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node 
> manager started
> 2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
> Configuration provider started
> 2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
> Checking file:flume.conf for changes
> 2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider: 
> Reloading configuration file:flume.conf
> 2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for 
> avro-forward-sink01: hdfs.fileType
> 2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks: 
> avro-forward-sink01 Agent: weblog-agent
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of 
> configuration for agent: weblog-agent, initial-configuration: 
> AgentCon

Re: Waiting for TLP web site before 1.2.0 release

2012-07-18 Thread Mike Percy 
As long as the generated HTML stays in the release tarball I'm fine with
adding PDF as an additional output format.


On Wed, Jul 18, 2012 at 4:14 PM, Ralph Goers  wrote:

> Also, I got a patch from the guy who makes the maven sphinx plugin to
> generate PDFs. I haven't tried it, but if it works and it isn't ugly I'd
> like to use it for the release docs.
>
> Sent from my iPad
>
> On Jul 18, 2012, at 3:11 PM, Mike Percy  wrote:
>
> > Ralph,
> > Nice work getting the site up. I cannot find the page with the unreadable
> > text - I looked on Firefox on Mac and it seems OK. Can you provide a
> link?
> > If we can fix that or if it's not a blocker than I will complete the
> > release and send the ANNOUNCE email this evening.
> >
> > Regards,
> > Mike
> >
> > On Wed, Jul 18, 2012 at 1:17 PM, Ralph Goers  >wrote:
> >
> >> The Flume site is live at http://flume.apache.org/, however I've
> noticed
> >> at least one problem - the text for releases on the release page is
> white
> >> on a white background so it looks like there is nothing there. If you
> >> actually click on them they change color and then you can see them. I
> will
> >> see if I can't fix that. The  content for releases 1.0.0, 1.1.0 and
> 1.2.0
> >> needs to be uploaded into the content area, if we have any.
> >>
> >> To change the web site all you have to do is check out
> >> https://svn.apache.org/repos/asf/flume/site/trunk and then edit
> whatever
> >> needs to be changed and commit it. The staging site will automatically
> >> rebuild.  Once you like it in the staging site it can be made live by
> using
> >> https://cms.apache.org/#bookmark and selecting Flume.
> >>
> >>
> >> Ralph
> >>
> >>
> >>
> >> On Jul 17, 2012, at 2:11 AM, Mike Percy wrote:
> >>
> >>> I just pushed the Maven artifacts to the staging repository. But I have
> >> not
> >>> yet promoted them to the Maven Central release repository yet. I'll do
> >> that
> >>> as part of the final step in the release process after the site is up.
> >>>
> >>> Regards,
> >>> Mike
> >>>
> >>>
> >>> On Mon, Jul 16, 2012 at 3:58 PM, Mike Percy  wrote:
> >>>
>  I will push the Maven artifacts to staging within the next couple of
> >> days.
> 
>  Regards,
>  Mike
> 
> 
>  On Mon, Jul 16, 2012 at 6:20 AM, Ralph Goers 
> wrote:
> 
> > I see no mention of the maven artifacts in the list below.
> >
> > Ralph
> >
> > On Jul 15, 2012, at 10:42 PM, Mike Percy  wrote:
> >
> >> Hi folks,
> >> Although the release vote is closed, I feel it would be good for the
> >> project to announce our first TLP release from our TLP web site
> >> http://flume.apache.org ... Infra is now in the process of moving
> the
> > site
> >> (see INFRA-4982), after which I'll push the bits to the Apache
> mirrors
> > to
> >> make them available for download, add the release info to the
> website,
> > and
> >> send an [ANNOUNCE] mail. Please let me know if you have concerns and
> > thanks
> >> for your patience in this matter.
> >>
> >> Regards,
> >> Mike
> >
> 
> 
> >>
> >>
>

Re: Waiting for TLP web site before 1.2.0 release

2012-07-18 Thread Mike Percy 
Unfortunately, distributing the 1.2.0 release is still blocked on one more
INFRA ticket, which I have just added a note to:
https://issues.apache.org/jira/browse/INFRA-4984


On Wed, Jul 18, 2012 at 4:09 PM, Ralph Goers  wrote:

> That is probably because I fixed it right after I sent the message below.
>
> Feel free to go ahead. Let me know if you have any problems.
>
> Ralph
>
> Sent from my iPad
>
> On Jul 18, 2012, at 3:11 PM, Mike Percy  wrote:
>
> > Ralph,
> > Nice work getting the site up. I cannot find the page with the unreadable
> > text - I looked on Firefox on Mac and it seems OK. Can you provide a
> link?
> > If we can fix that or if it's not a blocker than I will complete the
> > release and send the ANNOUNCE email this evening.
> >
> > Regards,
> > Mike
> >
> > On Wed, Jul 18, 2012 at 1:17 PM, Ralph Goers  >wrote:
> >
> >> The Flume site is live at http://flume.apache.org/, however I've
> noticed
> >> at least one problem - the text for releases on the release page is
> white
> >> on a white background so it looks like there is nothing there. If you
> >> actually click on them they change color and then you can see them. I
> will
> >> see if I can't fix that. The  content for releases 1.0.0, 1.1.0 and
> 1.2.0
> >> needs to be uploaded into the content area, if we have any.
> >>
> >> To change the web site all you have to do is check out
> >> https://svn.apache.org/repos/asf/flume/site/trunk and then edit
> whatever
> >> needs to be changed and commit it. The staging site will automatically
> >> rebuild.  Once you like it in the staging site it can be made live by
> using
> >> https://cms.apache.org/#bookmark and selecting Flume.
> >>
> >>
> >> Ralph
> >>
> >>
> >>
> >> On Jul 17, 2012, at 2:11 AM, Mike Percy wrote:
> >>
> >>> I just pushed the Maven artifacts to the staging repository. But I have
> >> not
> >>> yet promoted them to the Maven Central release repository yet. I'll do
> >> that
> >>> as part of the final step in the release process after the site is up.
> >>>
> >>> Regards,
> >>> Mike
> >>>
> >>>
> >>> On Mon, Jul 16, 2012 at 3:58 PM, Mike Percy  wrote:
> >>>
>  I will push the Maven artifacts to staging within the next couple of
> >> days.
> 
>  Regards,
>  Mike
> 
> 
>  On Mon, Jul 16, 2012 at 6:20 AM, Ralph Goers 
> wrote:
> 
> > I see no mention of the maven artifacts in the list below.
> >
> > Ralph
> >
> > On Jul 15, 2012, at 10:42 PM, Mike Percy  wrote:
> >
> >> Hi folks,
> >> Although the release vote is closed, I feel it would be good for the
> >> project to announce our first TLP release from our TLP web site
> >> http://flume.apache.org ... Infra is now in the process of moving
> the
> > site
> >> (see INFRA-4982), after which I'll push the bits to the Apache
> mirrors
> > to
> >> make them available for download, add the release info to the
> website,
> > and
> >> send an [ANNOUNCE] mail. Please let me know if you have concerns and
> > thanks
> >> for your patience in this matter.
> >>
> >> Regards,
> >> Mike
> >
> 
> 
> >>
> >>
>

[jira] [Updated] (FLUME-1382) Flume adopt message from existing local Scribe

2012-07-18 Thread Denny Ye (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denny Ye updated FLUME-1382:


Attachment: FLUME-1382.txt

ScribeSource in legacy sub-folder

> Flume adopt message from existing local Scribe
> --
>
> Key: FLUME-1382
> URL: https://issues.apache.org/jira/browse/FLUME-1382
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.2.0
>Reporter: Denny Ye
>Assignee: Denny Ye
>Priority: Minor
>  Labels: scribe, thrift
> Fix For: v1.3.0
>
> Attachments: FLUME-1382.txt
>
>
> Currently, we are using Scribe in data ingest system. Central Scribe is hard 
> to maintain and upgrade. Thus, we would like to replace central Scribe with 
> Flume and adopt message from existing and amounts of local Scribe. This can 
> be treated as legacy part.
> We have generated ScribeSource and used with more effective Thrift code 
> without deserializing.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (FLUME-1379) avro source , hdfs sink, no event writed to hdfs

2012-07-18 Thread jack li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jack li updated FLUME-1379:
---

Fix Version/s: (was: v1.3.0)
  Description: 
hello everyone!
1, source's type is exec  and sink's type is hdfs .the event can write to the 
hdfs.But when I change the source's type to avro ,no event write to hdfs.
2, flume.conf 
#List sources, sinks and channels in the agent
weblog-agent.sources = s1
weblog-agent.sinks = avro-forward-sink01
weblog-agent.channels = jdbc-channel01
#define the flow
#webblog-agent sources config
weblog-agent.sources.s1.channels = jdbc-channel01
weblog-agent.sources.s1.type = avro
weblog-agent.sources.s1.port = 41414
weblog-agent.sources.s1.bind = 0.0.0.0
#avro sink properties
weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
weblog-agent.sinks.avro-forward-sink01.type = hdfs
weblog-agent.sinks.avro-forward-sink01.hdfs.path = 
hdfs://hadoop-master.:9000/user/flume/webtest
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_%p
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_web001
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = fl...@hadoop.com
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab = 
/var/run/flume-ng/flume.keytab
weblog-agent.sinks.avro-forward-sink01.hdfs.maxOpenFiles = 65535
weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.batchSize = 3
weblog-agent.sinks.avro-forward-sink01.hdfs.txnEventMax = 2
weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream
#channels config
weblog-agent.channels.jdbc-channel01.type = memory
weblog-agent.channels.jdbc-channel01.capacity = 20
weblog-agent.channels.jdbc-channel01.transactionCapacity = 2

3,flume.log

2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 1
2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting - weblog-agent
2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node 
manager starting
2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 10
2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider: 
Configuration provider starting
2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node 
manager started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
Configuration provider started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
Checking file:flume.conf for changes
2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider: 
Reloading configuration file:flume.conf
2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for 
avro-forward-sink01: hdfs.fileType
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks: 
avro-forward-sink01 Agent: weblog-agent
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of 
configuration for agent: weblog-agent, initial-configuration: 
AgentConfiguration[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro, 
bind=0.0.0.0} }}
CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=2, 
capacity=20, type=memory} }}
SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream, 
hdfs.kerberosPrincipal=fl...@hadoop.com, hdfs.txnEventMax=2, 
hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab, 
hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=3, 
hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01, 
hdfs.rollCount=0} }}

2012-07-18 23:53:31,271 DEBUG conf.FlumeConfiguration: Created channel 
jdbc-channel01
2012-07-18 23:53:31,289 DEBUG conf.FlumeConfiguration: Creating sink: 
avro-forward-sink01 using HDFS
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Post validation 
configuration for weblog

Re: Review Request: FLUME-1358: Add a regex-based filtering interceptor

2012-07-18 Thread Patrick Wendell 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5898/
---

(Updated July 19, 2012, 2:54 a.m.)


Review request for Flume.


Changes
---

fixing small whitespace problem - this is still ready for review thanks.


Description
---

Adding a regex-based filtering interceptor


This addresses bug FLUME-1358.
https://issues.apache.org/jira/browse/FLUME-1358


Diffs (updated)
-

  flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java 
0c112d0 
  
flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexFilteringInterceptor.java
 PRE-CREATION 
  
flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexFilteringInterceptor.java
 PRE-CREATION 
  flume-ng-doc/sphinx/FlumeUserGuide.rst 7967d7c 

Diff: https://reviews.apache.org/r/5898/diff/


Testing
---

Unit tests are included


Thanks,

Patrick Wendell

[jira] [Created] (FLUME-1382) Flume adopt message from existing local Scribe

2012-07-18 Thread Denny Ye (JIRA) 
Denny Ye created FLUME-1382:
---

 Summary: Flume adopt message from existing local Scribe
 Key: FLUME-1382
 URL: https://issues.apache.org/jira/browse/FLUME-1382
 Project: Flume
  Issue Type: Improvement
  Components: Sinks+Sources
Affects Versions: v1.2.0
Reporter: Denny Ye
Assignee: Denny Ye
Priority: Minor
 Fix For: v1.3.0


Currently, we are using Scribe in data ingest system. Central Scribe is hard to 
maintain and upgrade. Thus, we would like to replace central Scribe with Flume 
and adopt message from existing and amounts of local Scribe. This can be 
treated as legacy part.

We have generated ScribeSource and used with more effective Thrift code without 
deserializing.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Waiting for TLP web site before 1.2.0 release

2012-07-18 Thread Ralph Goers 
Also, I got a patch from the guy who makes the maven sphinx plugin to generate 
PDFs. I haven't tried it, but if it works and it isn't ugly I'd like to use it 
for the release docs.

Sent from my iPad

On Jul 18, 2012, at 3:11 PM, Mike Percy  wrote:

> Ralph,
> Nice work getting the site up. I cannot find the page with the unreadable
> text - I looked on Firefox on Mac and it seems OK. Can you provide a link?
> If we can fix that or if it's not a blocker than I will complete the
> release and send the ANNOUNCE email this evening.
> 
> Regards,
> Mike
> 
> On Wed, Jul 18, 2012 at 1:17 PM, Ralph Goers 
> wrote:
> 
>> The Flume site is live at http://flume.apache.org/, however I've noticed
>> at least one problem - the text for releases on the release page is white
>> on a white background so it looks like there is nothing there. If you
>> actually click on them they change color and then you can see them. I will
>> see if I can't fix that. The  content for releases 1.0.0, 1.1.0 and 1.2.0
>> needs to be uploaded into the content area, if we have any.
>> 
>> To change the web site all you have to do is check out
>> https://svn.apache.org/repos/asf/flume/site/trunk and then edit whatever
>> needs to be changed and commit it. The staging site will automatically
>> rebuild.  Once you like it in the staging site it can be made live by using
>> https://cms.apache.org/#bookmark and selecting Flume.
>> 
>> 
>> Ralph
>> 
>> 
>> 
>> On Jul 17, 2012, at 2:11 AM, Mike Percy wrote:
>> 
>>> I just pushed the Maven artifacts to the staging repository. But I have
>> not
>>> yet promoted them to the Maven Central release repository yet. I'll do
>> that
>>> as part of the final step in the release process after the site is up.
>>> 
>>> Regards,
>>> Mike
>>> 
>>> 
>>> On Mon, Jul 16, 2012 at 3:58 PM, Mike Percy  wrote:
>>> 
 I will push the Maven artifacts to staging within the next couple of
>> days.
 
 Regards,
 Mike
 
 
 On Mon, Jul 16, 2012 at 6:20 AM, Ralph Goers  wrote:
 
> I see no mention of the maven artifacts in the list below.
> 
> Ralph
> 
> On Jul 15, 2012, at 10:42 PM, Mike Percy  wrote:
> 
>> Hi folks,
>> Although the release vote is closed, I feel it would be good for the
>> project to announce our first TLP release from our TLP web site
>> http://flume.apache.org ... Infra is now in the process of moving the
> site
>> (see INFRA-4982), after which I'll push the bits to the Apache mirrors
> to
>> make them available for download, add the release info to the website,
> and
>> send an [ANNOUNCE] mail. Please let me know if you have concerns and
> thanks
>> for your patience in this matter.
>> 
>> Regards,
>> Mike
> 
 
 
>> 
>>

Re: Waiting for TLP web site before 1.2.0 release

2012-07-18 Thread Ralph Goers 
That is probably because I fixed it right after I sent the message below.

Feel free to go ahead. Let me know if you have any problems.

Ralph

Sent from my iPad

On Jul 18, 2012, at 3:11 PM, Mike Percy  wrote:

> Ralph,
> Nice work getting the site up. I cannot find the page with the unreadable
> text - I looked on Firefox on Mac and it seems OK. Can you provide a link?
> If we can fix that or if it's not a blocker than I will complete the
> release and send the ANNOUNCE email this evening.
> 
> Regards,
> Mike
> 
> On Wed, Jul 18, 2012 at 1:17 PM, Ralph Goers 
> wrote:
> 
>> The Flume site is live at http://flume.apache.org/, however I've noticed
>> at least one problem - the text for releases on the release page is white
>> on a white background so it looks like there is nothing there. If you
>> actually click on them they change color and then you can see them. I will
>> see if I can't fix that. The  content for releases 1.0.0, 1.1.0 and 1.2.0
>> needs to be uploaded into the content area, if we have any.
>> 
>> To change the web site all you have to do is check out
>> https://svn.apache.org/repos/asf/flume/site/trunk and then edit whatever
>> needs to be changed and commit it. The staging site will automatically
>> rebuild.  Once you like it in the staging site it can be made live by using
>> https://cms.apache.org/#bookmark and selecting Flume.
>> 
>> 
>> Ralph
>> 
>> 
>> 
>> On Jul 17, 2012, at 2:11 AM, Mike Percy wrote:
>> 
>>> I just pushed the Maven artifacts to the staging repository. But I have
>> not
>>> yet promoted them to the Maven Central release repository yet. I'll do
>> that
>>> as part of the final step in the release process after the site is up.
>>> 
>>> Regards,
>>> Mike
>>> 
>>> 
>>> On Mon, Jul 16, 2012 at 3:58 PM, Mike Percy  wrote:
>>> 
 I will push the Maven artifacts to staging within the next couple of
>> days.
 
 Regards,
 Mike
 
 
 On Mon, Jul 16, 2012 at 6:20 AM, Ralph Goers  wrote:
 
> I see no mention of the maven artifacts in the list below.
> 
> Ralph
> 
> On Jul 15, 2012, at 10:42 PM, Mike Percy  wrote:
> 
>> Hi folks,
>> Although the release vote is closed, I feel it would be good for the
>> project to announce our first TLP release from our TLP web site
>> http://flume.apache.org ... Infra is now in the process of moving the
> site
>> (see INFRA-4982), after which I'll push the bits to the Apache mirrors
> to
>> make them available for download, add the release info to the website,
> and
>> send an [ANNOUNCE] mail. Please let me know if you have concerns and
> thanks
>> for your patience in this matter.
>> 
>> Regards,
>> Mike
> 
 
 
>> 
>>

[jira] [Created] (FLUME-1381) Error in website

2012-07-18 Thread Hari Shreedharan (JIRA) 
Hari Shreedharan created FLUME-1381:
---

 Summary: Error in website
 Key: FLUME-1381
 URL: https://issues.apache.org/jira/browse/FLUME-1381
 Project: Flume
  Issue Type: Task
Reporter: Hari Shreedharan
Priority: Minor


Website shows an error on this page: 
http://flume.apache.org/releases/1.2.0.html#version-1-2-0:
[FLUME-1200] - HDFSEventSink causes *.snappy file to be created in HDFS even 
when snappy isn’t used (due to missing lib)

System Message: WARNING/2 
(/usr/home/cmsslave/slave15/flume-site-staging/build/trunk/content/sphinx/releases/1.2.0.rst,
 line 122); backlink

Inline emphasis start-string without end-string.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [DISCUSS] Git as primary source control for Flume

2012-07-18 Thread Hari Shreedharan 
I filed an Infra ticket for the migration: 
https://issues.apache.org/jira/browse/INFRA-5065 

Thanks,
Hari

-- 
Hari Shreedharan


On Tuesday, July 17, 2012 at 1:22 PM, Hari Shreedharan wrote:

> Current projects using git as primary version control: 
> https://git-wip-us.apache.org/repos/asf 
> 
> 
> 
> -- 
> Hari Shreedharan
> 
> 
> On Sunday, July 15, 2012 at 12:46 PM, Hari Shreedharan wrote:
> 
> > I have stared a vote on this topic.  
> > 
> > -- 
> > Hari Shreedharan
> > 
> > 
> > On Friday, July 13, 2012 at 2:40 AM, Arvind Prabhakar wrote:
> > 
> > > +1 for using Git as primary source control system.
> > > 
> > > Thanks Hari for following up on this.
> > > 
> > > Regards,
> > > Arvind Prabhakar
> > > 
> > > On Wed, Jul 11, 2012 at 7:16 PM, Leslin  > > (mailto:leslin...@gmail.com)> wrote:
> > > 
> > > > +1 for this proposal. Git is fine for me. I never back to SVN after I
> > > > touched git.
> > > > 
> > > > 2012/7/12 Mike Percy mailto:mpe...@apache.org)>
> > > > 
> > > > > On Wed, Jul 11, 2012 at 5:45 PM, Ralph Goers 
> > > > > mailto:ralph.go...@dslextreme.com)
> > > > > > wrote:
> > > > > 
> > > > > 
> > > > > > IMO the person who wrote the code is the one who should get credit.
> > > > > 
> > > > > Of course they should get the credit for the work.
> > > > > 
> > > > > Anyone who has ever performed a careful code review knows that it can 
> > > > > be
> > > > > time-consuming work. I assume that's one reason why we currently list
> > > > > 
> > > > 
> > > > both
> > > > > the author and the committer in the commit message.
> > > > > 
> > > > > Regards,
> > > > > Mike
> > > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > --
> > > > 
> > > > 
> > > > 
> > > > Best Regards
> > > > 
> > > > Leslin 
> > 
>

[VOTE][RESULT] GIT as primary source control for Apache Flume

2012-07-18 Thread Hari Shreedharan 
The community vote to change the primary source control from Subversion to git 
for Apache Flume has passed 
with consensus from the PMC and more than two-thirds majority from the 
community.

Tally of votes:
15   +1 votes
0 +0 votes
1 -1 votes

+1 votes:
Will McQueen (PMC)
Mubarak Seyed
Brock Noland (PMC) 
Ralph Goers (PMC)
Harish Mandala
Hari Shreedharan (PMC)
Patrick Wendell
Mike Percy (PMC)
Raymond Ng 
Rajesh Koilpillai
Jarek Jarcec Cecho (PMC)
Arvind Prabhakar (PMC)
alo alt
Inder Pali
Alan Rodriguez


+0 votes:
--


-1 votes:
Mark Stern


Thanks once again to all who voted. 

Thanks,
Hari


-- 
Hari Shreedharan

Re: Waiting for TLP web site before 1.2.0 release

2012-07-18 Thread Mike Percy 
Ralph,
Nice work getting the site up. I cannot find the page with the unreadable
text - I looked on Firefox on Mac and it seems OK. Can you provide a link?
If we can fix that or if it's not a blocker than I will complete the
release and send the ANNOUNCE email this evening.

Regards,
Mike

On Wed, Jul 18, 2012 at 1:17 PM, Ralph Goers wrote:

> The Flume site is live at http://flume.apache.org/, however I've noticed
> at least one problem - the text for releases on the release page is white
> on a white background so it looks like there is nothing there. If you
> actually click on them they change color and then you can see them. I will
> see if I can't fix that. The  content for releases 1.0.0, 1.1.0 and 1.2.0
> needs to be uploaded into the content area, if we have any.
>
> To change the web site all you have to do is check out
> https://svn.apache.org/repos/asf/flume/site/trunk and then edit whatever
> needs to be changed and commit it. The staging site will automatically
> rebuild.  Once you like it in the staging site it can be made live by using
> https://cms.apache.org/#bookmark and selecting Flume.
>
>
> Ralph
>
>
>
> On Jul 17, 2012, at 2:11 AM, Mike Percy wrote:
>
> > I just pushed the Maven artifacts to the staging repository. But I have
> not
> > yet promoted them to the Maven Central release repository yet. I'll do
> that
> > as part of the final step in the release process after the site is up.
> >
> > Regards,
> > Mike
> >
> >
> > On Mon, Jul 16, 2012 at 3:58 PM, Mike Percy  wrote:
> >
> >> I will push the Maven artifacts to staging within the next couple of
> days.
> >>
> >> Regards,
> >> Mike
> >>
> >>
> >> On Mon, Jul 16, 2012 at 6:20 AM, Ralph Goers  wrote:
> >>
> >>> I see no mention of the maven artifacts in the list below.
> >>>
> >>> Ralph
> >>>
> >>> On Jul 15, 2012, at 10:42 PM, Mike Percy  wrote:
> >>>
>  Hi folks,
>  Although the release vote is closed, I feel it would be good for the
>  project to announce our first TLP release from our TLP web site
>  http://flume.apache.org ... Infra is now in the process of moving the
> >>> site
>  (see INFRA-4982), after which I'll push the bits to the Apache mirrors
> >>> to
>  make them available for download, add the release info to the website,
> >>> and
>  send an [ANNOUNCE] mail. Please let me know if you have concerns and
> >>> thanks
>  for your patience in this matter.
> 
>  Regards,
>  Mike
> >>>
> >>
> >>
>
>

Re: Review Request: FLUME-1380. File channel log write consistency

2012-07-18 Thread Hari Shreedharan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6038/#review9263
---

Ship it!


Ship It!

- Hari Shreedharan


On July 18, 2012, 10:04 p.m., Arvind Prabhakar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6038/
> ---
> 
> (Updated July 18, 2012, 10:04 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> ---
> 
> Fixed the log file writer to make a single call to the underlying channel in 
> order to serialize the event. 
> 
> 
> This addresses bug FLUME-1380.
> https://issues.apache.org/jira/browse/FLUME-1380
> 
> 
> Diffs
> -
> 
>   
> /trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
>  1363086 
> 
> Diff: https://reviews.apache.org/r/6038/diff/
> 
> 
> Testing
> ---
> 
> Ran all tests. Manually populated and flushed a file channel.
> 
> 
> Thanks,
> 
> Arvind Prabhakar
> 
>

[jira] [Updated] (FLUME-1380) File channel log can record the op code and not the operation in some cases

2012-07-18 Thread Arvind Prabhakar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arvind Prabhakar updated FLUME-1380:


Attachment: FLUME-1380-1.patch

> File channel log can record the op code and not the operation in some cases
> ---
>
> Key: FLUME-1380
> URL: https://issues.apache.org/jira/browse/FLUME-1380
> Project: Flume
>  Issue Type: Bug
>Reporter: Arvind Prabhakar
>Assignee: Arvind Prabhakar
> Attachments: FLUME-1380-1.patch
>
>
> There is a race condition in the system where the log file can record the 
> beginning of a record and be shutdown before the remaining record is written 
> out. This will lead to the system not starting up correctly again with 
> exceptions like:
> {noformat}
> ERROR file.Log: Failed to initialize Log
> java.io.IOException: Header 80808080 not expected value: deadbeef
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request: FLUME-1380. File channel log write consistency

2012-07-18 Thread Arvind Prabhakar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6038/
---

Review request for Flume.


Description
---

Fixed the log file writer to make a single call to the underlying channel in 
order to serialize the event. 


This addresses bug FLUME-1380.
https://issues.apache.org/jira/browse/FLUME-1380


Diffs
-

  
/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
 1363086 

Diff: https://reviews.apache.org/r/6038/diff/


Testing
---

Ran all tests. Manually populated and flushed a file channel.


Thanks,

Arvind Prabhakar

Re: [VOTE] GIT as primary source control for Apache Flume

2012-07-18 Thread Hari Shreedharan 
The VOTE is now closed. I will be sending a RESULT mail separately.

Thanks to all who voted.

Thanks,
Hari


-- 
Hari Shreedharan


On Sunday, July 15, 2012 at 12:22 PM, Hari Shreedharan wrote:

> This is a call for VOTE to change the primary source control for Apache Flume 
> from Subversion to Git. 
> The discussion thread for this topic is available at [1]. 
> 
> Please cast your votes:
> 
> [ ] +1 Accept git as the primary source control for Apache Flume.
> [ ] +0 Indifferent to the proposed change of source control from subversion 
> to git for Apache Flume.
> [ ] -1 Reject the proposed change of source control from subversion to git 
> for Apache Flume.
> 
> 
> 
> This vote will remain open for at least 72 hours. 
> 
> 
> [1] Discussion thread for the proposal:
> http://apache.markmail.org/thread/kg5ojiu4i2kypfjk
> 
> 
> Thanks,
> Hari Shreedharan
> 
> 
> -- 
> Hari Shreedharan
>

Re: Waiting for TLP web site before 1.2.0 release

2012-07-18 Thread Ralph Goers 
The Flume site is live at http://flume.apache.org/, however I've noticed at 
least one problem - the text for releases on the release page is white on a 
white background so it looks like there is nothing there. If you actually click 
on them they change color and then you can see them. I will see if I can't fix 
that. The  content for releases 1.0.0, 1.1.0 and 1.2.0 needs to be uploaded 
into the content area, if we have any.  

To change the web site all you have to do is check out 
https://svn.apache.org/repos/asf/flume/site/trunk and then edit whatever needs 
to be changed and commit it. The staging site will automatically rebuild.  Once 
you like it in the staging site it can be made live by using 
https://cms.apache.org/#bookmark and selecting Flume.


Ralph



On Jul 17, 2012, at 2:11 AM, Mike Percy wrote:

> I just pushed the Maven artifacts to the staging repository. But I have not
> yet promoted them to the Maven Central release repository yet. I'll do that
> as part of the final step in the release process after the site is up.
> 
> Regards,
> Mike
> 
> 
> On Mon, Jul 16, 2012 at 3:58 PM, Mike Percy  wrote:
> 
>> I will push the Maven artifacts to staging within the next couple of days.
>> 
>> Regards,
>> Mike
>> 
>> 
>> On Mon, Jul 16, 2012 at 6:20 AM, Ralph Goers  wrote:
>> 
>>> I see no mention of the maven artifacts in the list below.
>>> 
>>> Ralph
>>> 
>>> On Jul 15, 2012, at 10:42 PM, Mike Percy  wrote:
>>> 
 Hi folks,
 Although the release vote is closed, I feel it would be good for the
 project to announce our first TLP release from our TLP web site
 http://flume.apache.org ... Infra is now in the process of moving the
>>> site
 (see INFRA-4982), after which I'll push the bits to the Apache mirrors
>>> to
 make them available for download, add the release info to the website,
>>> and
 send an [ANNOUNCE] mail. Please let me know if you have concerns and
>>> thanks
 for your patience in this matter.
 
 Regards,
 Mike
>>> 
>> 
>>

[jira] [Created] (FLUME-1380) File channel log can record the op code and not the operation in some cases

2012-07-18 Thread Arvind Prabhakar (JIRA) 
Arvind Prabhakar created FLUME-1380:
---

 Summary: File channel log can record the op code and not the 
operation in some cases
 Key: FLUME-1380
 URL: https://issues.apache.org/jira/browse/FLUME-1380
 Project: Flume
  Issue Type: Bug
Reporter: Arvind Prabhakar
Assignee: Arvind Prabhakar


There is a race condition in the system where the log file can record the 
beginning of a record and be shutdown before the remaining record is written 
out. This will lead to the system not starting up correctly again with 
exceptions like:

{noformat}
ERROR file.Log: Failed to initialize Log
java.io.IOException: Header 80808080 not expected value: deadbeef
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (FLUME-1379) source's type is avro and sink's type is hdfs but sink don't to work

2012-07-18 Thread jack li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jack li updated FLUME-1379:
---

Description: 
hello everyone!
1, source's type is exec  and sink's type is hdfs .the event can write to the 
hdfs.But when I change the source's type is avro ,and no change sink's type,no 
event write to hdfs.
2, flume.conf 
#List sources, sinks and channels in the agent
weblog-agent.sources = s1
weblog-agent.sinks = avro-forward-sink01
weblog-agent.channels = jdbc-channel01
#define the flow
#webblog-agent sources config
weblog-agent.sources.s1.channels = jdbc-channel01
weblog-agent.sources.s1.type = avro
weblog-agent.sources.s1.port = 41414
weblog-agent.sources.s1.bind = 0.0.0.0
#avro sink properties
weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
weblog-agent.sinks.avro-forward-sink01.type = hdfs
weblog-agent.sinks.avro-forward-sink01.hdfs.path = 
hdfs://hadoop-master.:9000/user/flume/webtest
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_%p
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_web001
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = fl...@hadoop.com
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab = 
/var/run/flume-ng/flume.keytab
#weblog-agent.sinks.avro-forward-sink01.hdfs.rollInterval = 43200
# 3 hours#
weblog-agent.sinks.avro-forward-sink01.hdfs.maxOpenFiles = 65535
weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.batchSize = 3
weblog-agent.sinks.avro-forward-sink01.hdfs.txnEventMax = 2
weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream
#channels config
weblog-agent.channels.jdbc-channel01.type = memory
weblog-agent.channels.jdbc-channel01.capacity = 20
weblog-agent.channels.jdbc-channel01.transactionCapacity = 2

3,flume.log

2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 1
2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting - weblog-agent
2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node 
manager starting
2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 10
2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider: 
Configuration provider starting
2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node 
manager started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
Configuration provider started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
Checking file:flume.conf for changes
2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider: 
Reloading configuration file:flume.conf
2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for 
avro-forward-sink01: hdfs.fileType
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks: 
avro-forward-sink01 Agent: weblog-agent
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of 
configuration for agent: weblog-agent, initial-configuration: 
AgentConfiguration[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro, 
bind=0.0.0.0} }}
CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=2, 
capacity=20, type=memory} }}
SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream, 
hdfs.kerberosPrincipal=fl...@hadoop.com, hdfs.txnEventMax=2, 
hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab, 
hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=3, 
hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01, 
hdfs.rollCount=0} }}

2012-07-18 23:53:31,271 DEBUG conf.FlumeConfiguration: Created channel 
jdbc-channel01
2012-07-18 23:53:31,289 DEBUG conf.FlumeConfiguration: Creating sink: 
avro-forward-sink01 using HDFS
2012-07-18 23:53:31,292

[jira] [Updated] (FLUME-1379) source's type is avro and sink's type is hdfs but sink don't to work

2012-07-18 Thread jack li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jack li updated FLUME-1379:
---

Description: 
hello everyone!
1, source's type is exec  and sink's type is hdfs .the event can write to the 
hdfs.But when I change the source's type is avro ,and no change sink's type,no 
event write to hdfs.
2, flume.conf 
#List sources, sinks and channels in the agent
weblog-agent.sources = s1
weblog-agent.sinks = avro-forward-sink01
weblog-agent.channels = jdbc-channel01
#define the flow
#webblog-agent sources config
weblog-agent.sources.s1.channels = jdbc-channel01
weblog-agent.sources.s1.type = avro
weblog-agent.sources.s1.port = 41414
weblog-agent.sources.s1.bind = 0.0.0.0
#avro sink properties
weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
weblog-agent.sinks.avro-forward-sink01.type = hdfs
weblog-agent.sinks.avro-forward-sink01.hdfs.path = 
hdfs://hadoop-master.:9000/user/flume/webtest
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_%p
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_web001
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = fl...@hadoop.com
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab = 
/var/run/flume-ng/flume.keytab
weblog-agent.sinks.avro-forward-sink01.hdfs.maxOpenFiles = 65535
weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.batchSize = 3
weblog-agent.sinks.avro-forward-sink01.hdfs.txnEventMax = 2
weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream
#channels config
weblog-agent.channels.jdbc-channel01.type = memory
weblog-agent.channels.jdbc-channel01.capacity = 20
weblog-agent.channels.jdbc-channel01.transactionCapacity = 2

3,flume.log

2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 1
2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting - weblog-agent
2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node 
manager starting
2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 10
2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider: 
Configuration provider starting
2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node 
manager started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
Configuration provider started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
Checking file:flume.conf for changes
2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider: 
Reloading configuration file:flume.conf
2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for 
avro-forward-sink01: hdfs.fileType
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks: 
avro-forward-sink01 Agent: weblog-agent
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of 
configuration for agent: weblog-agent, initial-configuration: 
AgentConfiguration[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro, 
bind=0.0.0.0} }}
CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=2, 
capacity=20, type=memory} }}
SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream, 
hdfs.kerberosPrincipal=fl...@hadoop.com, hdfs.txnEventMax=2, 
hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab, 
hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=3, 
hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01, 
hdfs.rollCount=0} }}

2012-07-18 23:53:31,271 DEBUG conf.FlumeConfiguration: Created channel 
jdbc-channel01
2012-07-18 23:53:31,289 DEBUG conf.FlumeConfiguration: Creating sink: 
avro-forward-sink01 using HDFS
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Post validation 
configuration for weblog-agent
Agent

[jira] [Updated] (FLUME-1379) source's type is avro and sink's type is hdfs but sink don't to work

2012-07-18 Thread jack li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jack li updated FLUME-1379:
---

Description: 
hello everyone!
1, source's type is exec  and sink's type is hdfs .the event can write to the 
hdfs.But when I change the source's type is avro ,and no change sink's type,no 
event write to hdfs.
2, flume.conf 
#List sources, sinks and channels in the agent
weblog-agent.sources = s1
#weblog-agent.sources.s1.interceptors = ts host
#weblog-agent.sources.s1.interceptors.ts.type = 
org.apache.flume.interceptor.TimestampInterceptor$Builder
#weblog-agent.sources.s1.interceptors.host.type = 
org.apache.flume.interceptor.HostInterceptor$Builder
#weblog-agent.sources.s1.interceptors.host.useIP = false
#weblog-agent.sources.s1.interceptors.host.preserveExisting = true
weblog-agent.sinks = avro-forward-sink01
weblog-agent.channels = jdbc-channel01
#define the flow
#webblog-agent sources config
weblog-agent.sources.s1.channels = jdbc-channel01
weblog-agent.sources.s1.type = avro
weblog-agent.sources.s1.port = 41414
weblog-agent.sources.s1.bind = 0.0.0.0
#weblog-agent.sources.s1.restart = true
#weblog-agent.sources.s1.restartThrottle = 1000
#weblog-agent.sources.s1.command = tail -f /opt/apps/nginx/logs/access.log
#weblog-agent.sources.s1.selector.type = replicating
#avro sink properties
weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
weblog-agent.sinks.avro-forward-sink01.type = hdfs
#weblog-agent.sinks.avro-forward-sink01.hdfs.path = 
hdfs://hadoop-master.:9000/user/flume/webtest/%Y/%m/%d
weblog-agent.sinks.avro-forward-sink01.hdfs.path = 
hdfs://hadoop-master.:9000/user/flume/webtest
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_%p
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_web001
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = fl...@hadoop.com
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab = 
/var/run/flume-ng/flume.keytab
#weblog-agent.sinks.avro-forward-sink01.hdfs.rollInterval = 43200
# 3 hours#
weblog-agent.sinks.avro-forward-sink01.hdfs.maxOpenFiles = 65535
weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.batchSize = 3
weblog-agent.sinks.avro-forward-sink01.hdfs.txnEventMax = 2
weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream
#channels config
weblog-agent.channels.jdbc-channel01.type = memory
weblog-agent.channels.jdbc-channel01.capacity = 20
weblog-agent.channels.jdbc-channel01.transactionCapacity = 2

3,flume.log

2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 1
2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting - weblog-agent
2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node 
manager starting
2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 10
2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider: 
Configuration provider starting
2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node 
manager started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
Configuration provider started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
Checking file:flume.conf for changes
2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider: 
Reloading configuration file:flume.conf
2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for 
avro-forward-sink01: hdfs.fileType
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks: 
avro-forward-sink01 Agent: weblog-agent
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of 
configuration for agent: weblog-agent, initial-configuration: 
AgentConfiguration[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, typ

[jira] [Updated] (FLUME-1379) source's type is avro and sink's type is hdfs but sink don't to work

2012-07-18 Thread jack li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jack li updated FLUME-1379:
---

Description: 
hello everyone!
1, source's type is exec  and sink's type is hdfs .the event can write to the 
hdfs.But when I change the source's type is avro ,and no change sink's type,no 
event write to hdfs.
2, flume.conf 
#List sources, sinks and channels in the agent
weblog-agent.sources = tail
weblog-agent.sinks = avro-forward-sink01
weblog-agent.channels = jdbc-channel01
#webblog-agent sources config
weblog-agent.sources.tail.type = avro
weblog-agent.sources.tail.port = 41414
weblog-agent.sources.tail.bind = 0.0.0.0
#avro sink properties
weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
weblog-agent.sinks.avro-forward-sink01.type = hdfs
weblog-agent.sinks.avro-forward-sink01.hdfs.path = 
hdfs://hadoop-01:8020/flume/web
weblog-agent.sinks.avro-forward-sink01.hdfs.rollInterval = 120
#channels config
weblog-agent.channels.jdbc-channel01.type = memory

3,flume.log

2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 1
2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting - weblog-agent
2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node 
manager starting
2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 10
2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider: 
Configuration provider starting
2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node 
manager started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
Configuration provider started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: 
Checking file:flume.conf for changes
2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider: 
Reloading configuration file:flume.conf
2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for 
avro-forward-sink01: hdfs.fileType
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks: 
avro-forward-sink01 Agent: weblog-agent
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of 
configuration for agent: weblog-agent, initial-configuration: 
AgentConfiguration[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro, 
bind=0.0.0.0} }}
CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=2, 
capacity=20, type=memory} }}
SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream, 
hdfs.kerberosPrincipal=fl...@hadoop.com, hdfs.txnEventMax=2, 
hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab, 
hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=3, 
hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01, 
hdfs.rollCount=0} }}

2012-07-18 23:53:31,271 DEBUG conf.FlumeConfiguration: Created channel 
jdbc-channel01
2012-07-18 23:53:31,289 DEBUG conf.FlumeConfiguration: Creating sink: 
avro-forward-sink01 using HDFS
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Post validation 
configuration for weblog-agent
AgentConfiguration created without Configuration stubs for which only basic 
syntactical validation was performed[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro, 
bind=0.0.0.0} }}
CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=2, 
capacity=20, type=memory} }}
SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream, 
hdfs.kerberosPrincipal=fl...@hadoop.com, hdfs.txnEventMax=2, 
hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab, 
hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=3, 
hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01, 
hdfs.rollCount=0} }}

2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Channels:jdbc-channel01

2012-07-18 23:53:31,292 DEBUG conf.FlumeConfigura

OSCON

2012-07-18 Thread Ralph Goers 
Is anyone from Flume at OSCon?  If you are here I would love to say Hi!

Ralph

[jira] [Created] (FLUME-1379) source's type is avro and sink's type is hdfs but sink don't to work

2012-07-18 Thread jack li (JIRA) 
jack li created FLUME-1379:
--

 Summary: source's type  is avro and sink's type is hdfs but sink 
don't to work
 Key: FLUME-1379
 URL: https://issues.apache.org/jira/browse/FLUME-1379
 Project: Flume
  Issue Type: Bug
  Components: Sinks+Sources
Affects Versions: v1.1.0, v1.3.0
 Environment: rhel5.7 ,jdk1.6,yum installed flume-ng 
1.1.0+120-1.cdh4.0.0  ,mvn install 1.3.
Reporter: jack li
 Fix For: v1.3.0


hello everyone!
1, source's type is exec  and sink's type is hdfs .the event can write to the 
hdfs.But when I change the source's type is avro ,and no change sink's type,no 
event write to hdfs.
2, flume.conf 
#List sources, sinks and channels in the agent
weblog-agent.sources = tail
weblog-agent.sinks = avro-forward-sink01
weblog-agent.channels = jdbc-channel01
#webblog-agent sources config
weblog-agent.sources.tail.type = avro
weblog-agent.sources.tail.port = 41414
weblog-agent.sources.tail.bind = 0.0.0.0
#avro sink properties
weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
weblog-agent.sinks.avro-forward-sink01.type = hdfs
weblog-agent.sinks.avro-forward-sink01.hdfs.path = 
hdfs://hadoop-01:8020/flume/web
weblog-agent.sinks.avro-forward-sink01.hdfs.rollInterval = 120
#channels config
weblog-agent.channels.jdbc-channel01.type = memory

3,flume.log

2012-07-18 21:03:18,731 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 1
2012-07-18 21:03:18,734 INFO node.FlumeNode: Flume node starting - weblog-agent
2012-07-18 21:03:18,738 INFO nodemanager.DefaultLogicalNodeManager: Node 
manager starting
2012-07-18 21:03:18,738 INFO lifecycle.LifecycleSupervisor: Starting lifecycle 
supervisor 10
2012-07-18 21:03:18,738 INFO properties.PropertiesFileConfigurationProvider: 
Configuration provider starting
2012-07-18 21:03:18,740 INFO properties.PropertiesFileConfigurationProvider: 
Reloading configuration file:flume.conf
2012-07-18 21:03:18,748 INFO conf.FlumeConfiguration: Added sinks: 
avro-forward-sink01 Agent: weblog-agent
2012-07-18 21:03:18,748 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 21:03:18,748 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 21:03:18,748 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 21:03:18,748 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 21:03:18,748 INFO conf.FlumeConfiguration: 
Processing:avro-forward-sink01
2012-07-18 21:03:18,784 INFO conf.FlumeConfiguration: Post-validation flume 
configuration contains configuration  for agents: [weblog-agent]
2012-07-18 21:03:18,785 INFO properties.PropertiesFileConfigurationProvider: 
Creating channels
2012-07-18 21:03:18,857 INFO instrumentation.MonitoredCounterGroup: Monitoried 
counter group for type: CHANNEL, name: jdbc-channel01, registered successfully.
2012-07-18 21:03:18,857 INFO properties.PropertiesFileConfigurationProvider: 
created channel jdbc-channel01
2012-07-18 21:03:18,872 INFO instrumentation.MonitoredCounterGroup: Monitoried 
counter group for type: SOURCE, name: tail, registered successfully.
2012-07-18 21:03:18,886 INFO sink.DefaultSinkFactory: Creating instance of 
sink: avro-forward-sink01, type: hdfs
2012-07-18 21:03:19,142 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false
2012-07-18 21:03:19,144 INFO instrumentation.MonitoredCounterGroup: Monitoried 
counter group for type: SINK, name: avro-forward-sink01, registered 
successfully.
2012-07-18 21:03:19,146 INFO nodemanager.DefaultLogicalNodeManager: Starting 
new configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: { source:Avro 
source tail: { bindAddress: localhost, port: 41414 } }} 
sinkRunners:{avro-forward-sink01=SinkRunner: { 
policy:org.apache.flume.sink.DefaultSinkProcessor@57a7ddcf counterGroup:{ 
name:null counters:{} } }} 
channels:{jdbc-channel01=org.apache.flume.channel.MemoryChannel@4dd36dfe} }
2012-07-18 21:03:19,146 INFO nodemanager.DefaultLogicalNodeManager: Starting 
Channel jdbc-channel01
2012-07-18 21:03:19,147 INFO nodemanager.DefaultLogicalNodeManager: Waiting for 
channel: jdbc-channel01 to start. Sleeping for 500 ms
2012-07-18 21:03:19,147 INFO instrumentation.MonitoredCounterGroup: Component 
type: CHANNEL, name: jdbc-channel01 started
2012-07-18 21:03:19,648 INFO nodemanager.DefaultLogicalNodeManager: Starting 
Sink avro-forward-sink01
2012-07-18 21:03:19,649 INFO nodemanager.DefaultLogicalNodeManager: Starting 
Source tail
2012-07-18 21:03:19,650 INFO source.AvroSource: Starting Avro source tail: { 
bindAddress: localhost, port: 41414 }...
2012-07-18 21:03:19,656 INFO instrumentation.MonitoredCounterGroup: Component 
type: SINK, name: avro-forward-sink01 started
2012-07-18 21:03:20,096 INFO instrumentation.MonitoredCounterGroup: Component 
type: SOURCE, name: tail started
2012-07-18 21:03:20,096 INFO source.AvroSource: Avro source tail started.

 

--
This

[jira] [Commented] (FLUME-1378) Sink falls into infinite lifecycle with EventDeliveryException

2012-07-18 Thread Denny Ye (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417017#comment-13417017
 ] 

Denny Ye commented on FLUME-1378:
-

OK, it can be fixed with parameter checking. The impact of this bug is 
terrible, infinite lifecycle of Sink, and huge heap with full GC in 90% time, 
causes the 'GC overhead limit exceeded'

> Sink falls into infinite lifecycle with EventDeliveryException
> --
>
> Key: FLUME-1378
> URL: https://issues.apache.org/jira/browse/FLUME-1378
> Project: Flume
>  Issue Type: Bug
>  Components: Channel, Sinks+Sources
>Affects Versions: v1.2.0
>Reporter: Denny Ye
>Assignee: Denny Ye
> Fix For: v1.3.0
>
>
> I tested the Flume and found a critical issue. HDFSEventSink with 
> 'txnEventMax' parameter > MemoryChannel 'transactionCapacity' parameter, Sink 
> falls into infinite lifecycle and all Events has been hold in MemoryChannel 
> with error message:
> "Take list for MemoryTransaction, capacity  full, consider committing more 
> frequently,increasing capacity, or increasing thread count"

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1378) Sink falls into infinite lifecycle with EventDeliveryException

2012-07-18 Thread Mike Percy (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416991#comment-13416991
 ] 

Mike Percy commented on FLUME-1378:
---

Not sure this is a critical issue. Just a misconfiguration.

> Sink falls into infinite lifecycle with EventDeliveryException
> --
>
> Key: FLUME-1378
> URL: https://issues.apache.org/jira/browse/FLUME-1378
> Project: Flume
>  Issue Type: Bug
>  Components: Channel, Sinks+Sources
>Affects Versions: v1.2.0
>Reporter: Denny Ye
>Assignee: Denny Ye
> Fix For: v1.3.0
>
>
> I tested the Flume and found a critical issue. HDFSEventSink with 
> 'txnEventMax' parameter > MemoryChannel 'transactionCapacity' parameter, Sink 
> falls into infinite lifecycle and all Events has been hold in MemoryChannel 
> with error message:
> "Take list for MemoryTransaction, capacity  full, consider committing more 
> frequently,increasing capacity, or increasing thread count"

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (FLUME-1378) Sink falls into infinite lifecycle with EventDeliveryException

2012-07-18 Thread Denny Ye (JIRA) 
Denny Ye created FLUME-1378:
---

 Summary: Sink falls into infinite lifecycle with 
EventDeliveryException
 Key: FLUME-1378
 URL: https://issues.apache.org/jira/browse/FLUME-1378
 Project: Flume
  Issue Type: Bug
  Components: Channel, Sinks+Sources
Affects Versions: v1.2.0
Reporter: Denny Ye
Assignee: Denny Ye
 Fix For: v1.3.0


I tested the Flume and found a critical issue. HDFSEventSink with 'txnEventMax' 
parameter > MemoryChannel 'transactionCapacity' parameter, Sink falls into 
infinite lifecycle and all Events has been hold in MemoryChannel with error 
message:

"Take list for MemoryTransaction, capacity  full, consider committing more 
frequently,increasing capacity, or increasing thread count"

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: FLUME-1377: ChannelProcessor clobbers exception with NPE if Channel.getTransaction() throws

2012-07-18 Thread Hari Shreedharan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6029/#review9238
---

Ship it!


Ship It!

- Hari Shreedharan


On July 18, 2012, 7:41 a.m., Mike Percy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6029/
> ---
> 
> (Updated July 18, 2012, 7:41 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> ---
> 
> Ensure that ChannelProcessor carefully handles exceptions thrown from 
> getTransaction() to avoid clobbering them.
> 
> Also, avoid rethrowing RuntimeExceptions in AvroSource - instead just log 
> them and return FAILED.
> 
> 
> This addresses bug FLUME-1377.
> https://issues.apache.org/jira/browse/FLUME-1377
> 
> 
> Diffs
> -
> 
>   flume-ng-core/pom.xml 64e7f6b 
>   flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 
> 293b823 
>   flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 86e67b0 
>   
> flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelProcessor.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/6029/diff/
> 
> 
> Testing
> ---
> 
> Existing unit tests pass. Added 2 new unit tests.
> 
> 
> Thanks,
> 
> Mike Percy
> 
>

[jira] [Updated] (FLUME-1377) ChannelProcessor clobbers exception with NPE if Channel.getTransaction() throws

2012-07-18 Thread Mike Percy (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Percy updated FLUME-1377:
--

Attachment: FLUME-1377-1.patch

> ChannelProcessor clobbers exception with NPE if Channel.getTransaction() 
> throws
> ---
>
> Key: FLUME-1377
> URL: https://issues.apache.org/jira/browse/FLUME-1377
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.2.0
>Reporter: Mike Percy
>Assignee: Mike Percy
> Fix For: v1.3.0
>
> Attachments: FLUME-1377-1.patch
>
>
> If Channel.getTransaction() throws then that Throwable is masked by a 
> NullPointerException in the catch block of 
> ChannelProcessor.processEventBatch() ... so all we see is a 
> NullPointerException when we should be seeing the actual source of the error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request: FLUME-1377: ChannelProcessor clobbers exception with NPE if Channel.getTransaction() throws

2012-07-18 Thread Mike Percy 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6029/
---

Review request for Flume.


Description
---

Ensure that ChannelProcessor carefully handles exceptions thrown from 
getTransaction() to avoid clobbering them.

Also, avoid rethrowing RuntimeExceptions in AvroSource - instead just log them 
and return FAILED.


This addresses bug FLUME-1377.
https://issues.apache.org/jira/browse/FLUME-1377


Diffs
-

  flume-ng-core/pom.xml 64e7f6b 
  flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 
293b823 
  flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 86e67b0 
  
flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelProcessor.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/6029/diff/


Testing
---

Existing unit tests pass. Added 2 new unit tests.


Thanks,

Mike Percy

[jira] [Created] (FLUME-1377) ChannelProcessor clobbers exception with NPE if Channel.getTransaction() throws

2012-07-18 Thread Mike Percy (JIRA) 
Mike Percy created FLUME-1377:
-

 Summary: ChannelProcessor clobbers exception with NPE if 
Channel.getTransaction() throws
 Key: FLUME-1377
 URL: https://issues.apache.org/jira/browse/FLUME-1377
 Project: Flume
  Issue Type: Bug
Reporter: Mike Percy
Assignee: Mike Percy
 Fix For: v1.3.0


If Channel.getTransaction() throws then that Throwable is masked by a 
NullPointerException in the catch block of ChannelProcessor.processEventBatch() 
... so all we see is a NullPointerException when we should be seeing the actual 
source of the error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (FLUME-1377) ChannelProcessor clobbers exception with NPE if Channel.getTransaction() throws

2012-07-18 Thread Mike Percy (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Percy updated FLUME-1377:
--

Affects Version/s: v1.2.0

> ChannelProcessor clobbers exception with NPE if Channel.getTransaction() 
> throws
> ---
>
> Key: FLUME-1377
> URL: https://issues.apache.org/jira/browse/FLUME-1377
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.2.0
>Reporter: Mike Percy
>Assignee: Mike Percy
> Fix For: v1.3.0
>
>
> If Channel.getTransaction() throws then that Throwable is masked by a 
> NullPointerException in the catch block of 
> ChannelProcessor.processEventBatch() ... so all we see is a 
> NullPointerException when we should be seeing the actual source of the error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira