Re: [ANNOUNCE] Apache Flume 1.6.0 released

[Santiago Mola]

Congrats and thank you!

Best,

Santiago

2015-06-01 23:55 GMT+02:00 Johny Rufus :

> The Apache Flume team is pleased to announce the release of Flume
> version 1.6.0.
>
> Flume is a distributed, reliable, and available service for efficiently
> collecting, aggregating, and moving large amounts of log data.
>
> This release can be downloaded from the Flume download page at:
> http://flume.apache.org/download.html
>
> The change log and documentation are available on the 1.6.0 release page:
> http://flume.apache.org/releases/1.6.0.html
>
> Your help and feedback is more than welcome. For more information on how
> to report problems and to get involved, visit the project website at
> http://flume.apache.org/
>
> The Apache Flume Team
>



-- 

Santiago M. Mola



Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
*

Re: [DISCUSS] Drop Hadoop 1 support

[Santiago Mola]

+1

It's time to move on. Trying to support hadoop-1 is also a dependency hell
for plugins.

2015-05-13 5:39 GMT+02:00 Hari Shreedharan :

> Adding user@
>
> On Tue, May 12, 2015 at 5:56 PM, Ashish  wrote:
>
>> +1 (speaking as dev, it keeps life a bit simple)
>>
>> Would wait for the users to give +1's as they get directly affected
>> and possibly a few may have to upgrade Data Infrastructure.
>>
>>
>> On Wed, May 13, 2015 at 4:00 AM, Hari Shreedharan
>>  wrote:
>> > Flume devs,
>> >
>> > With Flume moving away from Java 6 in Flume 1.7, I would like to start
>> a discussion on whether we should completely drop Hadoop-1 support. I am
>> not sure we can even link against Hadoop-1 if we build with Java 7 as the
>> minimum requirement. Even if we can, I think it might be a good idea to
>> drop Hadoop 1 support at this point.
>> >
>> > Comments?
>> >
>> > Thanks,
>> > Hari Shreedharan
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>
>


-- 

Santiago M. Mola



Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
*

Re: Review Request 30675: Add Cassandra sink

[Santiago Mola]

> On Feb. 27, 2015, 6:22 a.m., Hari Shreedharan wrote:
> > I didn't do a deep dive, but I have one question - if you look at the HBase 
> > sink or HDFS sink, a pluggable serializer allows the user to convert Flume 
> > events into an HBase or HDFS friendly form. Is something like this not 
> > required in case of this sink? Looks like the current implementation 
> > depends on data in the headers to decide where to put the data. 
> > Configuration is a better way to achieve this, so I think having a 
> > serializer (+ a good default one which possibly looks at the headers?) 
> > would make sense, no?
> 
> Ashish Paliwal wrote:
> Agree for the need of Serializer here. Without it, we might be storing 
> the event as a blob in C* rt? Let me spend some more time with review

Alright! I would make the current behaviour the default serializer. Storing the 
event as a blob without any extra information (e.g. primary key) will probably 
not be possible. The approach we are following for default Cassandra/JDBC/etc 
serializers is trying to automatically map event headers to target schema, i.e. 
any event header with a name matching a field in the schema will be converted 
to an appropriate data type and inserted. This works for most of our use cases 
(we use interceptors in the source to transform each event an populate headers) 
and for corner cases we use a serializer with template queries, allowing the 
user to specify an arbitrary statement with some placeholders for headers or 
body.

I'll upload a new version ASAP with the serializers and some additional 
functionality (inserting to multiple tables at once).


- Santiago


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30675/#review74452
---


On Feb. 5, 2015, 5:19 p.m., Santiago Mola wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30675/
> ---
> 
> (Updated Feb. 5, 2015, 5:19 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2616
> https://issues.apache.org/jira/browse/FLUME-2616
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Add Cassandra sink
> 
> 
> Diffs
> -
> 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 7a1dfce 
>   flume-ng-sinks/flume-cassandra-sink/pom.xml PRE-CREATION 
>   
> flume-ng-sinks/flume-cassandra-sink/src/main/java/org/apache/flume/sink/cassandra/CassandraSink.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-cassandra-sink/src/main/java/org/apache/flume/sink/cassandra/CassandraTable.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-cassandra-sink/src/main/java/org/apache/flume/sink/cassandra/CassandraUtils.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/CassandraDataTypesIT.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/CassandraSinkIT.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/CassandraTestHelper.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/TestCassandraSink.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/TestCassandraTable.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/TestCassandraUtils.java
>  PRE-CREATION 
>   flume-ng-sinks/flume-cassandra-sink/src/test/resources/log4j.properties 
> PRE-CREATION 
>   flume-ng-sinks/pom.xml 4bac019 
>   pom.xml 1350fa4 
> 
> Diff: https://reviews.apache.org/r/30675/diff/
> 
> 
> Testing
> ---
> 
> Extensive unit tests and integration tests.
> 
> 
> Thanks,
> 
> Santiago Mola
> 
>

Review Request 31195: Improve RpcClient unit tests

[Santiago Mola]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31195/
---

Review request for Flume.


Bugs: FLUME-2629
https://issues.apache.org/jira/browse/FLUME-2629


Repository: flume-git


Description
---

Improve RpcClient unit tests


Diffs
-

  flume-ng-sdk/pom.xml 113b286 
  flume-ng-sdk/src/main/java/org/apache/flume/api/FailoverRpcClient.java 
db6905a 
  flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java 
3661672 
  flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientFactory.java 11bc94c 
  flume-ng-sdk/src/main/java/org/apache/flume/api/ThriftRpcClient.java 4f75a2b 
  flume-ng-sdk/src/main/java/org/apache/flume/util/OrderSelector.java fd9e81f 
  flume-ng-sdk/src/main/java/org/apache/flume/util/RandomOrderSelector.java 
df6fce9 
  flume-ng-sdk/src/test/java/org/apache/flume/api/RpcClientAnswer.java 
PRE-CREATION 
  flume-ng-sdk/src/test/java/org/apache/flume/api/RpcTestUtils.java 8806860 
  flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 
64dc181 
  
flume-ng-sdk/src/test/java/org/apache/flume/api/TestLoadBalancingRpcClient.java 
5d6828b 
  flume-ng-sdk/src/test/java/org/apache/flume/api/TestRpcClientFactory.java 
2899925 
  flume-ng-sdk/src/test/java/org/apache/flume/api/TestThriftRpcClient.java 
a8baaa8 
  flume-ng-sdk/src/test/java/org/apache/flume/api/ThriftTestingSource.java 
70d2c1b 
  flume-ng-sdk/src/test/java/org/apache/flume/util/TestOrderSelector.java 
PRE-CREATION 
  pom.xml ea7ffe3 

Diff: https://reviews.apache.org/r/31195/diff/


Testing
---

Test pass. Coverage is increased.


Thanks,

Santiago Mola

Review Request 30675: Add Cassandra sink

[Santiago Mola]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30675/
---

Review request for Flume.


Bugs: FLUME-2616
https://issues.apache.org/jira/browse/FLUME-2616


Repository: flume-git


Description
---

Add Cassandra sink


Diffs
-

  flume-ng-doc/sphinx/FlumeUserGuide.rst 7a1dfce 
  flume-ng-sinks/flume-cassandra-sink/pom.xml PRE-CREATION 
  
flume-ng-sinks/flume-cassandra-sink/src/main/java/org/apache/flume/sink/cassandra/CassandraSink.java
 PRE-CREATION 
  
flume-ng-sinks/flume-cassandra-sink/src/main/java/org/apache/flume/sink/cassandra/CassandraTable.java
 PRE-CREATION 
  
flume-ng-sinks/flume-cassandra-sink/src/main/java/org/apache/flume/sink/cassandra/CassandraUtils.java
 PRE-CREATION 
  
flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/CassandraDataTypesIT.java
 PRE-CREATION 
  
flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/CassandraSinkIT.java
 PRE-CREATION 
  
flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/CassandraTestHelper.java
 PRE-CREATION 
  
flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/TestCassandraSink.java
 PRE-CREATION 
  
flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/TestCassandraTable.java
 PRE-CREATION 
  
flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/TestCassandraUtils.java
 PRE-CREATION 
  flume-ng-sinks/flume-cassandra-sink/src/test/resources/log4j.properties 
PRE-CREATION 
  flume-ng-sinks/pom.xml 4bac019 
  pom.xml 1350fa4 

Diff: https://reviews.apache.org/r/30675/diff/


Testing
---

Extensive unit tests and integration tests.


Thanks,

Santiago Mola

Re: Using a PositionTracker in EventDeserializers

[Santiago Mola]

Hi,

2014-12-04 1:19 GMT+01:00 Hari Shreedharan :

> OK, I think I understand. Can you file a jira and post this info + a
> design doc if you have a design in mind?
>

Sure. I'll prepare a more elaborate proposal.

Best,
-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *

Review Request 28654: Remove unneeded repository declarations in pom.xml

[Santiago Mola]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28654/
---

Review request for Flume.


Bugs: FLUME-2567
https://issues.apache.org/jira/browse/FLUME-2567


Repository: flume-git


Description
---

Remove unneeded repository declarations in pom.xml


Diffs
-

  flume-ng-sinks/flume-dataset-sink/pom.xml e929d60 
  flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml 01e9f45 
  pom.xml 7d4f01a 

Diff: https://reviews.apache.org/r/28654/diff/


Testing
---

Builds local, builds on Travis CI.


Thanks,

Santiago Mola

Re: Using a PositionTracker in EventDeserializers

[Santiago Mola]

2014-12-02 7:04 GMT+01:00 Hari Shreedharan :

> Wouldn’t the mark and reset be enough? Do you really need access to the
> underlying offsets? The resettable stream already provides mark and reset
>

As far as I know, that is not enough. I'll explain some use cases and maybe
you can suggest a better approach:

XML deserialization


XPath and XQuery parsers require you to parse the whole input document
before traversing it. XPath can actually be streamed, but resuming the
stream from an arbitrary point is far from trivial. As far as I know,
streaming XQuery remains a research issue without any standard solution.

The solution we have implemented: read the whole document (.mark()'ing at
the begginning of the resettable stream) and extract all the events at
once. We have to track the event index we returned last, so we use a
PositionTracker where we actually store event index, not stream position.
If the source is re-created after a crash, it will start reading the file
from the beginning, but it will start returning events from the last event
index that is stored in the PositionTracker.

We're working with similar approaches for some internal projects such as
PDF deserialization.

Decompression
=

Resuming decompression of a compressed stream at an arbitrary input offset
(the one stored at resettable stream) is usually not possible. Also, there
is no way to map an arbitrary offset in the decompressed stream to an
offset in the compressed stream. So we apply the same mechanism as in the
previous case (but at the ResettableInputStream level): we use a
ResettableInputStream implementation that wraps another
ResettableInputStream. The DecompressInputStream marks the underlying
ResettableInputStream at 0 and starts decompressing, tracking the offset in
the decompressed stream. If resuming is needed, it starts decompressing
from the beginning, but skipping all input until the last offset tracked in
the decompressed stream. This approach is suboptimal, since it requires
using a buffer that is always as large as the maximum batch size in bytes,
but it works otherwise.

Best,
-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *

Re: Review Request 28532: FLUME-2524 Adding an HTTP Sink

[Santiago Mola]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28532/#review63346
---


Just a couple of comments:

- Inconsistent naming. It should be "maxConnections", not "maxconnections", not 
"max_connections". The same applies to every other property. I would also avoid 
arbitrary abbreviatures such as "evt" in "evt_form_key".
- Ideally, the way the HTTP request is formed should be moved out to a 
pluggable serializer class. This way, the sink can be extended without 
rewriting it in full. You can see examples in other sources and sinks.

- Santiago Mola


On Nov. 28, 2014, 7:11 p.m., Jeff Guilmard wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28532/
> ---
> 
> (Updated Nov. 28, 2014, 7:11 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> FLUME-2524 Adding an HTTP Sink
> 
> Related to JIRA FLUME-2524
> 
> Flume whould have an HTTP Sink, with following capacities:
> Using up to date performant Http Client
> Capacity to load balance on multiple target servers (simple round robin)
> Handle HTTP Authentication
> use HTTP POST
> Capacity to send binary data.
> 
> New patch, replacing the first one i posted, but i didn't ask for a review of 
> first one because of the perf issues.
> 
> This patch looks stable to me. It improves a lot the performances, which was 
> a major work in order to tune HttpClient.
> It adds also the monitoring part that was missing.
> This version is not accepting Batch of events (more than 1), as HTTP is not 
> transactional. I was not sure on how to handle this properly without breaking 
> the transaction behavior of Flume Channels. Therefore i did not work on that, 
> even if that could improve performances even more. Could be a future 
> improvement.
> 
> I've included a patch for the documentation (user guide), as it is a new sink 
> to configure.
> 
> 
> Diffs
> -
> 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 
> e3aedeb1937404c260a1b81e7f9746a1a2a511d3 
>   flume-ng-sinks/flume-http-sink/pom.xml PRE-CREATION 
>   
> flume-ng-sinks/flume-http-sink/src/main/java/org/apache/flume/sink/http/HTTPSink.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-http-sink/src/test/java/org/apache/flume/sink/http/TestHTTPSink.java
>  PRE-CREATION 
>   flume-ng-sinks/pom.xml 4bac01916b39e25ba8df03eb7d699ba5c639299e 
> 
> Diff: https://reviews.apache.org/r/28532/diff/
> 
> 
> Testing
> ---
> 
> Basic testing JUnit proposed in the patch.
> I've also spent a lot of time to test the performances of the sink, before 
> managing to get a correct version.
> 
> This patch is now used in production in the company I work for.
> 
> 
> Thanks,
> 
> Jeff Guilmard
> 
>

Using a PositionTracker in EventDeserializers

[Santiago Mola]

Hi,

We have a recurring need of Flume deserializers that go beyond line or
blob. Some examples are XML deserialization where events are generated with
XPath/XQuery expressions, parsers for XLS, PDF, etc.

There is no proper solution in Flume for these use case. A significant
amount of our projects required workarounds for this such as an external
preprocessing or postprocessing step.

So we have explored the following solutions to the problem:

- Using BlobDeserializer and then using an interceptor (1 to N events) to
perform the transformation. This is currently not possible since an
interceptor must output 0 or 1 event for each input event. This was brought
up in this mailing list long time ago [1] but it seems no one came up with
a viable solution.

- Implementing an EventDeserializer. We have done this in some cases with
different degrees of success. For example, with a XML deserializer with
XPath [2]. The main limitation of this approach is the lack of a common
method for position tracking at the deserializer level. Currently, Flume's
core has a PositionTracker at the Source/InputStream level, which tracks
the input offset. LineDeserializer and BlobDeserializer rely on the
assumption that events can be mapped to an input offset (i.e. an event can
be created by reading only from a given input offset). This assumption is
not valid for more complex use cases (e.g. can't produce events without
reading file headers). This can be solved by using a second PositionTracker
at the deserializer level. Here's a commit with a possible implementation
of this approach [3].

Do you think this is a problem worth solving in Flume? If yes, what would
be the best approach?


[1]
http://mail-archives.apache.org/mod_mbox/flume-dev/201208.mbox/%3CCABCB9rJ0-puRp1FfPfvyfO41wnMgUh=tifcpgufwxbnyv_p...@mail.gmail.com%3E
[2]
https://github.com/Stratio/flume-ingestion/tree/develop/stratio-deserializers/stratio-xmlxpath-deserializer
[3]
https://github.com/Stratio/flume/commit/a6fac7247b7fc48dec5dc3ab4c658ab4e5c0e753

Best,
-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *

Re: any patches need attention ?

[Santiago Mola]

Hi Roshan,

Here are some pending issues with submitted patches (sorted by descending
relevance):

https://issues.apache.org/jira/browse/FLUME-2487
https://issues.apache.org/jira/browse/FLUME-2516
https://issues.apache.org/jira/browse/FLUME-2488
https://issues.apache.org/jira/browse/FLUME-2480

Thanks!

2014-11-18 1:29 GMT+01:00 Roshan Naik :

> I have some time this week to review a few patches. if you have any patches
> needing a committer's attention let me know.
>
> -roshan
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *

Re: [ANNOUNCE] New Flume PMC Member - Roshan Naik

[Santiago Mola]

Congrats Roshan!

2014-11-04 23:12 GMT+01:00 Arvind Prabhakar :

> On behalf of Apache Flume PMC, it is my pleasure to announce that Roshan
> Naik has been elected to the Flume Project Management Committee. Roshan has
> been active with the project for many years and has been a committer on the
> project since September of 2013.
>
> Please join me in congratulating Roshan and welcoming him to the Flume PMC.
>
> Regards,
> Arvind Prabhakar
>



-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *

Review Request 27378: ResettableFileInputStream can't support ucs-4 character

[Santiago Mola]

20974 
  
flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/BlobDeserializer.java
 12bdc409ed1499c93c733c7ec8a9593c817738c0 
  
flume-ng-sinks/flume-ng-morphline-solr-sink/src/test/java/org/apache/flume/sink/solr/morphline/ResettableTestStringInputStream.java
 e6ee9b9ecffd49fd51ef698f51850ccc3525dc40 
  
flume-ng-sinks/flume-ng-morphline-solr-sink/src/test/java/org/apache/flume/sink/solr/morphline/SeekableByteArrayInputStream.java
 PRE-CREATION 
  
flume-ng-sinks/flume-ng-morphline-solr-sink/src/test/java/org/apache/flume/sink/solr/morphline/TestBlobDeserializer.java
 6172c6810d7d0f766087c4695f6ce9dac25ef091 

Diff: https://reviews.apache.org/r/27378/diff/


Testing
---

Passes tests, agents work.


Thanks,

Santiago Mola

Review Request 27325: Fix rounding error in SyslogParser

[Santiago Mola]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27325/
---

Review request for Flume.


Repository: flume-git


Description
---

Fix rounding error in SyslogParser


Diffs
-

  flume-ng-core/src/main/java/org/apache/flume/source/SyslogParser.java 
01713096c2c05fb19d38fc36d468f1ef04b81798 
  flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogParser.java 
6e0fd665c498a186f1058bafd3005d341fb68bc5 

Diff: https://reviews.apache.org/r/27325/diff/


Testing
---

Passdes tests


Thanks,

Santiago Mola

Re: Review Request 27325: Fix rounding error in SyslogParser

[Santiago Mola]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27325/
---

(Updated Oct. 29, 2014, 10:51 a.m.)


Review request for Flume.


Bugs: FLUME-2487
https://issues.apache.org/jira/browse/FLUME-2487


Repository: flume-git


Description
---

Fix rounding error in SyslogParser


Diffs
-

  flume-ng-core/src/main/java/org/apache/flume/source/SyslogParser.java 
01713096c2c05fb19d38fc36d468f1ef04b81798 
  flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogParser.java 
6e0fd665c498a186f1058bafd3005d341fb68bc5 

Diff: https://reviews.apache.org/r/27325/diff/


Testing
---

Passdes tests


Thanks,

Santiago Mola

Review Request 27306: TestElasticSearchRestClient fails on Oracle JDK 8

[Santiago Mola]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27306/
---

Review request for Flume.


Bugs: FLUME-2488
https://issues.apache.org/jira/browse/FLUME-2488


Repository: flume-git


Description
---

TestElasticSearchRestClient fails on Oracle JDK 8


Diffs
-

  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java
 0d1c37f 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/client/TestElasticSearchRestClient.java
 b7d8822 

Diff: https://reviews.apache.org/r/27306/diff/


Testing
---

Test passes.


Thanks,

Santiago Mola

Re: Contribution of a new tailing source

[Santiago Mola]

2014-10-09 8:22 GMT+02:00 飯島賢志 :

> Can we contribute it to the Apache Flume community?
> If acceptable I will attach the patch of it in JIRA.
>

That would be definitely useful for us.

Best,
-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *

Integration tests

[Santiago Mola]

Hi all,

I have been experiencing problems when using embedded software (HBase,
Cassandra) in unit tests. My main itches are:

1) Dependencies incompatibilities (e.g. Guava). Now this could be
considered a blocker: https://issues.apache.org/jira/browse/FLUME-2286

2) Cross-polluting the classpath for both Flume and the embedded server.
When pulling a big project like HBase or Cassandra to the test classpath,
we're testing an environment that is different from both Flume's and the
embedded software. This can create test bugs and, what is worse, mask bugs
that won't show up in test but will show up in production.

3) Testing different versions of HBase or Cassandra aggravates the problem.

What do you think about moving these tests to integration-test (with JUnit
to keep it simple) and set them up to use external instances of required
services?

Best,
-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *

Re: Test failures

[Santiago Mola]

Hi Hari,

2014-10-03 7:36 GMT+02:00 Hari Shreedharan :

> We don’t yet support JDK 8. I don’t think any real testing has been done
> to build against JDK 8. We should prioritize fixing the JDK 7 failures
> first.
>

Ok. I'll continue creating JIRAs for JDK 8 just to identify problems in
advance, but feel free to ignore them.

Best,
-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *

Test failures

[Santiago Mola]

Hi,

I'm trying to get a Flume version that passes tests on Travis CI with
OpenJDK 7, Oracle JDK 7 and Oracle JDK 8 [1]. Currently, *sometimes* it can
complete tests with Oracle JDK 7. There're still a number of intermittent
test failures on JDK 7 and permanent failures on JDK 8. Here's a list of
JIRA's I've opened along the way:

Patch suggested:
https://issues.apache.org/jira/browse/FLUME-2474
https://issues.apache.org/jira/browse/FLUME-2483
https://issues.apache.org/jira/browse/FLUME-2486


No patch:
https://issues.apache.org/jira/browse/FLUME-2482
https://issues.apache.org/jira/browse/FLUME-2484 (Kafka)
https://issues.apache.org/jira/browse/FLUME-2485 (JDK 8)
https://issues.apache.org/jira/browse/FLUME-2488 (JDK 8?)
https://issues.apache.org/jira/browse/FLUME-2487

[1] https://travis-ci.org/Stratio/flume

Best,
-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *

Re: Major bugs when system locale is not en_US

[Santiago Mola]

Thanks Hari!

2014-10-01 19:08 GMT+02:00 Hari Shreedharan :

> I will look into these today
>
>
> Thanks,
> Hari
>
> On Wed, Oct 1, 2014 at 10:03 AM, Santiago Mola  wrote:
>
> > Hi,
> > There are multiple bugs when system locale is other than en_US. Here's a
> > list of patches fixing them:
> > https://issues.apache.org/jira/browse/FLUME-2475
> > https://issues.apache.org/jira/browse/FLUME-2472
> > https://issues.apache.org/jira/browse/FLUME-2481
> > Best,
> > --
> > Santiago M. Mola
> > <http://www.stratio.com/>
> > Avenida de Europa, 26. Ática 5. 3ª Planta
> > 28224 Pozuelo de Alarcón, Madrid
> > Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>*




-- 

Santiago M. Mola


<http://www.stratio.com/>
Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>*

Major bugs when system locale is not en_US

[Santiago Mola]

Hi,

There are multiple bugs when system locale is other than en_US. Here's a
list of patches fixing them:

https://issues.apache.org/jira/browse/FLUME-2475
https://issues.apache.org/jira/browse/FLUME-2472
https://issues.apache.org/jira/browse/FLUME-2481

Best,
-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *

Review Request 26140: Use Locale.ENGLISH for every toUpperCase/toLowerCase call

[Santiago Mola]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26140/
---

Review request for Flume.


Bugs: FLUME-2475
https://issues.apache.org/jira/browse/FLUME-2475


Repository: flume-git


Description
---

Use Locale.ENGLISH for every toUpperCase/toLowerCase call. This prevents bugs 
when the default system locale is not en_US.


Diffs
-

  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderFactory.java
 d5e9353 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderFactory.java
 60d51ed 
  
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/DatabaseType.java
 e26cb84 
  
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/TransactionIsolation.java
 ed157cf 
  
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestDatabaseTypeEnum.java
 55ad1eb 
  
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestTransactionIsolationLevelEnum.java
 ea0bac3 
  
flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAvroHeaders.java
 08a7203 
  
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java
 c867a03 
  
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
 f214149 
  
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java
 13c3360 
  
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java
 998c8c0 
  
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 
06e1763 
  
flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
 0bc3f23 
  
flume-ng-core/src/main/java/org/apache/flume/instrumentation/MonitoredCounterGroup.java
 1d3f0f1 
  
flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorBuilderFactory.java
 50e6fbc 
  
flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java
 e44978f 
  
flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java
 80b9cc3 
  
flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java
 dbb3935 
  
flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java
 d8e5ae2 
  flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java 
0e49e0b 
  flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 
1499872 
  flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java 
95cf13d 
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 
d0c2e99 
  flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 208fefe 
  flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java 
b12571b 
  flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java 
3beec06 
  
flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgentConfiguration.java
 06ea7c0 
  flume-ng-node/src/main/java/org/apache/flume/node/Application.java 832285a 
  flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientFactory.java 00d9cf0 
  
flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/DatasetSink.java
 4cd3027 
  
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AvroEventSerializer.java
 4b8fc78 
  
flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java
 addd97a 
  flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java f886c89 

Diff: https://reviews.apache.org/r/26140/diff/


Testing
---

Passes tests correctly.


Thanks,

Santiago Mola

Re: What Source/Sink would you want next?

[Santiago Mola]

Hi Arvind, hi Hari

2014-09-26 20:06 GMT+02:00 Arvind Prabhakar :

> Santiago (and others who would like to contribute) - please go ahead and
> create the necessary Jiras if they do not exist already, and invite the
> community to vote on those. That way we can prioritize the review and
> commit for functionality that is aligned with community requirements.
>


2014-09-26 20:10 GMT+02:00 Hari Shreedharan :

> Strong +1. If there are new features or fixes you want to contribute -
> please do, I am trying to make sure I spend some time reviewing and
> committing. If you don’t get a review in a few days, please ping on the
> jira - I will look at it!
>

Sure. I'll do. Thanks!

Best,
-- 

Santiago M. Mola



Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd *