[jira] [Commented] (FLUME-2458) Separate hdfs tmp directory for flume hdfs sink
[ https://issues.apache.org/jira/browse/FLUME-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496138#comment-14496138 ] Dylan Jones commented on FLUME-2458: Hello, Think I have found a workaround for now that allows this. If you overload the filePrefix with a path i.e. hdfs.path = /flume hdfs.filePrefix = data/ hdfs.fileSuffix = .avro hdfs.inUsePrefix = tmp/ Real files: /flume/data/123456.avro Temp files: /flume/tmp/data/123456.avro.tmp You can then use the location /flume/data for external tables and the likes for map reduce jobs Separate hdfs tmp directory for flume hdfs sink --- Key: FLUME-2458 URL: https://issues.apache.org/jira/browse/FLUME-2458 Project: Flume Issue Type: Improvement Components: Sinks+Sources Affects Versions: v1.5.0.1 Reporter: Sverre Bakke Assignee: Johny Rufus Priority: Minor The current HDFS sink will write temporary files to the same directory as the final file will be stored. This is a problem for several reasons: 1) File moving When mapreduce fetches a list of files to be processed and then processes files that are then gone (i.e. are moved from .tmp to whatever final name it is suppose to have), then the mapreduce job will crash. 2) File type When mapreduce decides how to process files, then it looks at files extension. If using compressed files, then it will decompress it for you. If the file has a .tmp file extension (in the same folder) then it will treat a compressed file as an uncompressed files, thus breaking the results of the mapreduce job. I propose that the sink gets an optional tmp path for storing these files to avoid these issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2672) NPE in KafkaSourceCounter
[ https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Paliwal updated FLUME-2672: -- Summary: NPE in KafkaSourceCounter (was: NPE in KafkaSinkCounter) NPE in KafkaSourceCounter - Key: FLUME-2672 URL: https://issues.apache.org/jira/browse/FLUME-2672 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.6.0 Environment: Mac OS 10.10.3, Java 1.7.0_60 Reporter: Rigo MacTaggart Priority: Trivial Labels: easyfix Fix For: v1.6.0 Attachments: FLUME-2672.patch Original Estimate: 0h Remaining Estimate: 0h A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() because it expects MonitoredCounterGroup.counterMap to contain key source.kafka.empty.count. A patch is included which adds this key to the ATTRIBUTES string array, which is used to pre-populate keys with an initial value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2672) NPE in KafkaSinkCounter
[ https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rigo MacTaggart updated FLUME-2672: --- Attachment: FLUME-2672.patch NPE in KafkaSinkCounter --- Key: FLUME-2672 URL: https://issues.apache.org/jira/browse/FLUME-2672 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.6.0 Environment: Mac OS 10.10.3, Java 1.7.0_60 Reporter: Rigo MacTaggart Priority: Trivial Labels: easyfix Fix For: v1.6.0 Attachments: FLUME-2672.patch Original Estimate: 0h Remaining Estimate: 0h A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() because it expects MonitoredCounterGroup.counterMap to contain key source.kafka.empty.count. A patch is included which adds this key to the ATTRIBUTES string array, which is used to pre-populate keys with an initial value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2672) NPE in KafkaSourceCounter
[ https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rigo MacTaggart updated FLUME-2672: --- Attachment: FLUME-2672-with-test.patch Added a test for KafkaSourceCounter. The test exercises the increment/getter methods. NPE in KafkaSourceCounter - Key: FLUME-2672 URL: https://issues.apache.org/jira/browse/FLUME-2672 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.6.0 Environment: Mac OS 10.10.3, Java 1.7.0_60 Reporter: Rigo MacTaggart Priority: Trivial Labels: easyfix Fix For: v1.6.0 Attachments: FLUME-2672-with-test.patch, FLUME-2672.patch Original Estimate: 0h Remaining Estimate: 0h A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() because it expects MonitoredCounterGroup.counterMap to contain key source.kafka.empty.count. A patch is included which adds this key to the ATTRIBUTES string array, which is used to pre-populate keys with an initial value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2672) NPE in KafkaSourceCounter
[ https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496248#comment-14496248 ] Ashish Paliwal commented on FLUME-2672: --- +1, Patch looks good. May be we should have some test cases around counters as well. NPE in KafkaSourceCounter - Key: FLUME-2672 URL: https://issues.apache.org/jira/browse/FLUME-2672 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.6.0 Environment: Mac OS 10.10.3, Java 1.7.0_60 Reporter: Rigo MacTaggart Priority: Trivial Labels: easyfix Fix For: v1.6.0 Attachments: FLUME-2672.patch Original Estimate: 0h Remaining Estimate: 0h A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() because it expects MonitoredCounterGroup.counterMap to contain key source.kafka.empty.count. A patch is included which adds this key to the ATTRIBUTES string array, which is used to pre-populate keys with an initial value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2672) NPE in KafkaSinkCounter
Rigo MacTaggart created FLUME-2672: -- Summary: NPE in KafkaSinkCounter Key: FLUME-2672 URL: https://issues.apache.org/jira/browse/FLUME-2672 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.6.0 Environment: Mac OS 10.10.3, Java 1.7.0_60 Reporter: Rigo MacTaggart Priority: Trivial Fix For: v1.6.0 A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() because it expects MonitoredCounterGroup.counterMap to contain key source.kafka.empty.count. A patch is included which adds this key to the ATTRIBUTES string array, which is used to pre-populate keys with an initial value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 33215: FLUME-2649 ElasticSearch sink should index nested JSON
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33215/#review80205
---
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
https://reviews.apache.org/r/33215/#comment129990
This should not only check if it's null, but also if it's a json content
type.
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
https://reviews.apache.org/r/33215/#comment129991
You don't have to create a tmp builder. To make sure the JSON is valid, you
just have to do:
while(parser.nextToken() != null) { }
- Francis Page
On April 15, 2015, 9:56 a.m., Benjamin Fiorini wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33215/
---
(Updated April 15, 2015, 9:56 a.m.)
Review request for Flume.
Repository: flume-git
Description
---
Elasticsearch sink fails to index malformed nested JSON
Diffs
-
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
de0acf4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
2f8fd6d
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java
78e1665
Diff: https://reviews.apache.org/r/33215/diff/
Testing
---
Thanks,
Benjamin Fiorini
[jira] [Updated] (FLUME-2670) Modify dependencies to not pull hadoop/hive related jars
[ https://issues.apache.org/jira/browse/FLUME-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johny Rufus updated FLUME-2670: --- Attachment: FLUME-2670.patch Made the following changes 1 - made flume-ng-auth hadoop-common dependency as optional 2 - made hive-exec and hive-metastore dependecy in kite dataset sink as optional 3 - removed unused import introduced in TestLog.java, one of the recent commits that depends on hadoop-common [~hshreedharan], can you let me know if the patch is fine, and making the above as optional is fine Modify dependencies to not pull hadoop/hive related jars Key: FLUME-2670 URL: https://issues.apache.org/jira/browse/FLUME-2670 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2670.patch Recent changes to kite-sdk and flume-ng-auth modules, have introduced hadoop and hive dependency jars to be pulled in the flume distribution. Working/testing on a patch, will upload soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 33215: FLUME-2649 ElasticSearch sink should index nested JSON
On April 15, 2015, 6:41 p.m., Francis Page wrote:
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java,
line 64
https://reviews.apache.org/r/33215/diff/1/?file=929718#file929718line64
You don't have to create a tmp builder. To make sure the JSON is valid,
you just have to do:
while(parser.nextToken() != null) { }
True, cause that's exactly what copyCurrentStructure does under the hood
(recursively).
On April 15, 2015, 6:41 p.m., Francis Page wrote:
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java,
line 44
https://reviews.apache.org/r/33215/diff/1/?file=929718#file929718line44
This should not only check if it's null, but also if it's a json
content type.
Not sure about this one, if it's JSON/YAML/SMILE, then contentType is not going
to be null and the data can be parsed as a complex field. IMHO this does the
job well.
- Benjamin
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33215/#review80205
---
On April 15, 2015, 11:56 a.m., Benjamin Fiorini wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33215/
---
(Updated April 15, 2015, 11:56 a.m.)
Review request for Flume.
Repository: flume-git
Description
---
Elasticsearch sink fails to index malformed nested JSON
Diffs
-
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
de0acf4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
2f8fd6d
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java
78e1665
Diff: https://reviews.apache.org/r/33215/diff/
Testing
---
Thanks,
Benjamin Fiorini
Re: Review Request 33215: FLUME-2649 ElasticSearch sink should index nested JSON
On April 15, 2015, 4:41 p.m., Francis Page wrote: flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java, line 44 https://reviews.apache.org/r/33215/diff/1/?file=929718#file929718line44 This should not only check if it's null, but also if it's a json content type. Benjamin Fiorini wrote: Not sure about this one, if it's JSON/YAML/SMILE, then contentType is not going to be null and the data can be parsed as a complex field. IMHO this does the job well. You're right, forget about this! - Francis --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33215/#review80205 --- On April 15, 2015, 9:56 a.m., Benjamin Fiorini wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33215/ --- (Updated April 15, 2015, 9:56 a.m.) Review request for Flume. Repository: flume-git Description --- Elasticsearch sink fails to index malformed nested JSON Diffs - flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java de0acf4 flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java 2f8fd6d flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java 78e1665 Diff: https://reviews.apache.org/r/33215/diff/ Testing --- Thanks, Benjamin Fiorini
Re: Flume 1.6 jiras
For tail source we definitely have to go with Java 7, so +1, for having a 1.7 release soon, with tail and s3 sources. Thanks, Rufus On Tue, Apr 14, 2015 at 4:27 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: I realize that a lot of you wanted the tail source and s3 source. I really don’t have the bandwidth to get it reviewed right now and I would like to get the Flume 1.6 release out soon. We can come back and do a 1.7 release with Java 7 as minimum required version in a month or something, and we can get both these in by then? Thanks, Hari On Sun, Apr 12, 2015 at 10:11 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: I think Johnny is working on it now, so I'd expect an RC put in the next few days! Thanks, Hari On Sun, Apr 12, 2015 at 3:43 AM, 李响 wate...@gmail.com wrote: Hi Johny and harry, I think it is a must to include FLUME-2575 https://issues.apache.org/jira/browse/FLUME-2575. Do you agree ? On Sat, Apr 11, 2015 at 11:21 AM, Ashish paliwalash...@gmail.com wrote: This jira FLUME-2632 - High CPU on KafkaSink, might also be worth adding to 1.6 It's a simple fix. On Tue, Apr 7, 2015 at 7:21 AM, Ashish paliwalash...@gmail.com wrote: If I am not late for the party, here are some JIRA's needing some attention FLUME-2613 - Tool/script for deleting individual message from queue FLUME-2589 - Add Related Projects Page FLUME-2413 - Add a new metric for the number of bytes transferred on a flume channel On Tue, Apr 7, 2015 at 2:43 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: I am committing those one by one (you will see a few today). Sorry for the delay! Thanks, Hari On Mon, Apr 6, 2015 at 2:11 PM, Roshan Naik ros...@hortonworks.com wrote: Johny, Earlier in this email thread you will find multiple responses to the same question posed by Hari. -roshan On 4/6/15 2:05 PM, Johny Rufus jru...@cloudera.com wrote: Folks, Am starting the release work for 1.6, can you reply back if there any pending patches that need to be committed for 1.6. Thanks Rufus On Mon, Apr 6, 2015 at 10:55 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: I am all in for including it, but it looks like it is Java 7+ - so we have to wait for Flume 1.7 to actually do it, since 1.6 will be the last release we will be supporting Java 6. I am ok with doing a 1.7 release right after. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal -- 李响 手机 cellphone ：+86-1368-113-8972 E-mail ：wate...@gmail.com MSN ：wate...@hotmail.com
[jira] [Assigned] (FLUME-2670) Modify dependencies to not pull hadoop/hive related jars
[ https://issues.apache.org/jira/browse/FLUME-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan reassigned FLUME-2670: --- Assignee: Hari Shreedharan (was: Johny Rufus) Modify dependencies to not pull hadoop/hive related jars Key: FLUME-2670 URL: https://issues.apache.org/jira/browse/FLUME-2670 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Hari Shreedharan Fix For: 1.6 Attachments: FLUME-2670.patch Recent changes to kite-sdk and flume-ng-auth modules, have introduced hadoop and hive dependency jars to be pulled in the flume distribution. Working/testing on a patch, will upload soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2670) Modify dependencies to not pull hadoop/hive related jars
[ https://issues.apache.org/jira/browse/FLUME-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan reassigned FLUME-2670: --- Assignee: Johny Rufus (was: Hari Shreedharan) Sorry, assigned to myself by mistake. Modify dependencies to not pull hadoop/hive related jars Key: FLUME-2670 URL: https://issues.apache.org/jira/browse/FLUME-2670 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2670.patch Recent changes to kite-sdk and flume-ng-auth modules, have introduced hadoop and hive dependency jars to be pulled in the flume distribution. Working/testing on a patch, will upload soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2670) Modify dependencies to not pull hadoop/hive related jars
[ https://issues.apache.org/jira/browse/FLUME-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497083#comment-14497083 ] ASF subversion and git services commented on FLUME-2670: Commit 636cff349a0012c2fd56525d0a69eb5a95610ea4 in flume's branch refs/heads/flume-1.6 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=636cff3 ] FLUME-2670. Modify dependencies to not pull hadoop/hive related jars (Johny Rufus via Hari) Modify dependencies to not pull hadoop/hive related jars Key: FLUME-2670 URL: https://issues.apache.org/jira/browse/FLUME-2670 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2670.patch Recent changes to kite-sdk and flume-ng-auth modules, have introduced hadoop and hive dependency jars to be pulled in the flume distribution. Working/testing on a patch, will upload soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2670) Modify dependencies to not pull hadoop/hive related jars
[ https://issues.apache.org/jira/browse/FLUME-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497082#comment-14497082 ] Johny Rufus commented on FLUME-2670: W/o the change to TestLog.java, the build was failing Modify dependencies to not pull hadoop/hive related jars Key: FLUME-2670 URL: https://issues.apache.org/jira/browse/FLUME-2670 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2670.patch Recent changes to kite-sdk and flume-ng-auth modules, have introduced hadoop and hive dependency jars to be pulled in the flume distribution. Working/testing on a patch, will upload soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Jenkins build is unstable: Flume-trunk-hbase-98 #92
See https://builds.apache.org/job/Flume-trunk-hbase-98/92/changes
[jira] [Commented] (FLUME-2673) Remove unused import in TestLog
[ https://issues.apache.org/jira/browse/FLUME-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497309#comment-14497309 ] Hudson commented on FLUME-2673: --- UNSTABLE: Integrated in Flume-trunk-hbase-98 #92 (See [https://builds.apache.org/job/Flume-trunk-hbase-98/92/]) FLUME-2673. Remove unused import in TestLog (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.gita=commith=364752a43f8ad7f85fd232351a179db017cd2265) * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java Remove unused import in TestLog Key: FLUME-2673 URL: https://issues.apache.org/jira/browse/FLUME-2673 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2673.patch There is an unused import in TestLog, that depends on javax.ws.rs, which causes compilation to fail when using mvn install -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2670) Modify dependencies to not pull hadoop/hive related jars
[ https://issues.apache.org/jira/browse/FLUME-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497086#comment-14497086 ] Hari Shreedharan commented on FLUME-2670: - that should be tracked separately. We should not be pulling in unrelated changes into the same commit. Modify dependencies to not pull hadoop/hive related jars Key: FLUME-2670 URL: https://issues.apache.org/jira/browse/FLUME-2670 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2670.patch Recent changes to kite-sdk and flume-ng-auth modules, have introduced hadoop and hive dependency jars to be pulled in the flume distribution. Working/testing on a patch, will upload soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2673) Remove unused import in TestLog
[ https://issues.apache.org/jira/browse/FLUME-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johny Rufus updated FLUME-2673: --- Attachment: FLUME-2673.patch Attaching patch, removing the unused import and fixing the dependency issue Remove unused import in TestLog Key: FLUME-2673 URL: https://issues.apache.org/jira/browse/FLUME-2673 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2673.patch There is an unused import in TestLog, that depends on javax.ws.rs, which causes compilation to fail when using mvn install -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Build failed in Jenkins: Flume-trunk-hbase-98 #91
See https://builds.apache.org/job/Flume-trunk-hbase-98/91/changes Changes: [hshreedharan] FLUME-2670. Modify dependencies to not pull hadoop/hive related jars -- [...truncated 9783 lines...] [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-channels/flume-spillable-memory-channel/pom.xml to org.apache.flume.flume-ng-channels/flume-spillable-memory-channel/1.6.0-SNAPSHOT/flume-spillable-memory-channel-1.6.0-SNAPSHOT.pom Sending artifact delta relative to Flume-trunk-hbase-98 » Flume NG Spillable Memory channel #90 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 3209 bytes Compression is 0.0% Took 0.15 sec [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-dist/pom.xml to org.apache.flume/flume-ng-dist/1.6.0-SNAPSHOT/flume-ng-dist-1.6.0-SNAPSHOT.pom Sending artifact delta relative to Flume-trunk-hbase-98 » Flume NG distribution #90 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 7022 bytes Compression is 0.0% Took 8.1 sec [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sinks/flume-dataset-sink/pom.xml to org.apache.flume.flume-ng-sinks/flume-dataset-sink/1.6.0-SNAPSHOT/flume-dataset-sink-1.6.0-SNAPSHOT.pom [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sinks/flume-dataset-sink/target/flume-dataset-sink-1.6.0-SNAPSHOT.jar to org.apache.flume.flume-ng-sinks/flume-dataset-sink/1.6.0-SNAPSHOT/flume-dataset-sink-1.6.0-SNAPSHOT.jar Sending artifact delta relative to Flume-trunk-hbase-98 » Flume NG Kite Dataset Sink #90 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 43067 bytes Compression is 0.0% Took 64 ms [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-legacy-sources/pom.xml to org.apache.flume/flume-ng-legacy-sources/1.6.0-SNAPSHOT/flume-ng-legacy-sources-1.6.0-SNAPSHOT.pom Sending artifact delta relative to Flume-trunk-hbase-98 » Flume legacy Sources #90 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 1752 bytes Compression is 0.0% Took 46 ms [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-channels/flume-kafka-channel/pom.xml to org.apache.flume.flume-ng-channels/flume-kafka-channel/1.6.0-SNAPSHOT/flume-kafka-channel-1.6.0-SNAPSHOT.pom [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-channels/flume-kafka-channel/target/flume-kafka-channel-1.6.0-SNAPSHOT.jar to org.apache.flume.flume-ng-channels/flume-kafka-channel/1.6.0-SNAPSHOT/flume-kafka-channel-1.6.0-SNAPSHOT.jar Sending artifact delta relative to Flume-trunk-hbase-98 » flume-kafka-channel #90 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 23886 bytes Compression is 0.0% Took 37 ms [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sources/flume-twitter-source/pom.xml to org.apache.flume.flume-ng-sources/flume-twitter-source/1.6.0-SNAPSHOT/flume-twitter-source-1.6.0-SNAPSHOT.pom [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sources/flume-twitter-source/target/flume-twitter-source-1.6.0-SNAPSHOT.jar to org.apache.flume.flume-ng-sources/flume-twitter-source/1.6.0-SNAPSHOT/flume-twitter-source-1.6.0-SNAPSHOT.jar Sending artifact delta relative to Flume-trunk-hbase-98 » Flume Twitter Source #90 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 17042 bytes Compression is 0.0% Took 69 ms [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-clients/pom.xml to org.apache.flume/flume-ng-clients/1.6.0-SNAPSHOT/flume-ng-clients-1.6.0-SNAPSHOT.pom Sending artifact delta relative to Flume-trunk-hbase-98 » Flume NG Clients #90 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 1548 bytes Compression is 0.0% Took 0.11 sec [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-channels/flume-jdbc-channel/pom.xml to org.apache.flume.flume-ng-channels/flume-jdbc-channel/1.6.0-SNAPSHOT/flume-jdbc-channel-1.6.0-SNAPSHOT.pom [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-channels/flume-jdbc-channel/target/flume-jdbc-channel-1.6.0-SNAPSHOT.jar to org.apache.flume.flume-ng-channels/flume-jdbc-channel/1.6.0-SNAPSHOT/flume-jdbc-channel-1.6.0-SNAPSHOT.jar Sending artifact delta relative to Flume-trunk-hbase-98 » Flume NG JDBC channel #90 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 56507 bytes Compression is 0.0% Took 0.11 sec [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml to org.apache.flume.flume-ng-sinks/flume-ng-morphline-solr-sink/1.6.0-SNAPSHOT/flume-ng-morphline-solr-sink-1.6.0-SNAPSHOT.pom [JENKINS] Archiving
[jira] [Commented] (FLUME-2658) Upgrade to Hbase 1.0
[ https://issues.apache.org/jira/browse/FLUME-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497156#comment-14497156 ] Roshan Naik commented on FLUME-2658: [~hshreedharan] Looks fine. Did a quick check by building against both 1.x and hbase-1 profiles and ran the hbase unit tests. Please go ahead. Upgrade to Hbase 1.0 Key: FLUME-2658 URL: https://issues.apache.org/jira/browse/FLUME-2658 Project: Flume Issue Type: Bug Reporter: Roshan Naik Assignee: Roshan Naik Attachments: FLUME-2658-2.patch, FLUME-2658-3.patch, FLUME-2658.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2658) Upgrade to Hbase 1.0
[ https://issues.apache.org/jira/browse/FLUME-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497286#comment-14497286 ] ASF subversion and git services commented on FLUME-2658: Commit 6a64c3bdfc52826a38ce65ad0a270f485fdb09bd in flume's branch refs/heads/flume-1.6 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=6a64c3b ] FLUME-2658. Upgrade to Hbase 1.0 (Roshan via Hari) Upgrade to Hbase 1.0 Key: FLUME-2658 URL: https://issues.apache.org/jira/browse/FLUME-2658 Project: Flume Issue Type: Bug Reporter: Roshan Naik Assignee: Roshan Naik Attachments: FLUME-2658-2.patch, FLUME-2658-3.patch, FLUME-2658.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2658) Upgrade to Hbase 1.0
[ https://issues.apache.org/jira/browse/FLUME-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497285#comment-14497285 ] ASF subversion and git services commented on FLUME-2658: Commit 019ab5207f2d2f0e7b9113b855b37d20dce664bf in flume's branch refs/heads/trunk from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=019ab52 ] FLUME-2658. Upgrade to Hbase 1.0 (Roshan via Hari) Upgrade to Hbase 1.0 Key: FLUME-2658 URL: https://issues.apache.org/jira/browse/FLUME-2658 Project: Flume Issue Type: Bug Reporter: Roshan Naik Assignee: Roshan Naik Attachments: FLUME-2658-2.patch, FLUME-2658-3.patch, FLUME-2658.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Flume 1.6 jiras
Once we branch for 1.6 release (making trunk Java 7+ only), I will review the tail source Thanks, Hari On Wed, Apr 15, 2015 at 11:14 AM, Johny Rufus jru...@cloudera.com wrote: For tail source we definitely have to go with Java 7, so +1, for having a 1.7 release soon, with tail and s3 sources. Thanks, Rufus On Tue, Apr 14, 2015 at 4:27 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: I realize that a lot of you wanted the tail source and s3 source. I really don’t have the bandwidth to get it reviewed right now and I would like to get the Flume 1.6 release out soon. We can come back and do a 1.7 release with Java 7 as minimum required version in a month or something, and we can get both these in by then? Thanks, Hari On Sun, Apr 12, 2015 at 10:11 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: I think Johnny is working on it now, so I'd expect an RC put in the next few days! Thanks, Hari On Sun, Apr 12, 2015 at 3:43 AM, 李响 wate...@gmail.com wrote: Hi Johny and harry, I think it is a must to include FLUME-2575 https://issues.apache.org/jira/browse/FLUME-2575. Do you agree ? On Sat, Apr 11, 2015 at 11:21 AM, Ashish paliwalash...@gmail.com wrote: This jira FLUME-2632 - High CPU on KafkaSink, might also be worth adding to 1.6 It's a simple fix. On Tue, Apr 7, 2015 at 7:21 AM, Ashish paliwalash...@gmail.com wrote: If I am not late for the party, here are some JIRA's needing some attention FLUME-2613 - Tool/script for deleting individual message from queue FLUME-2589 - Add Related Projects Page FLUME-2413 - Add a new metric for the number of bytes transferred on a flume channel On Tue, Apr 7, 2015 at 2:43 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: I am committing those one by one (you will see a few today). Sorry for the delay! Thanks, Hari On Mon, Apr 6, 2015 at 2:11 PM, Roshan Naik ros...@hortonworks.com wrote: Johny, Earlier in this email thread you will find multiple responses to the same question posed by Hari. -roshan On 4/6/15 2:05 PM, Johny Rufus jru...@cloudera.com wrote: Folks, Am starting the release work for 1.6, can you reply back if there any pending patches that need to be committed for 1.6. Thanks Rufus On Mon, Apr 6, 2015 at 10:55 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: I am all in for including it, but it looks like it is Java 7+ - so we have to wait for Flume 1.7 to actually do it, since 1.6 will be the last release we will be supporting Java 6. I am ok with doing a 1.7 release right after. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal -- 李响 手机 cellphone ：+86-1368-113-8972 E-mail ：wate...@gmail.com MSN ：wate...@hotmail.com
[jira] [Commented] (FLUME-2670) Modify dependencies to not pull hadoop/hive related jars
[ https://issues.apache.org/jira/browse/FLUME-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497080#comment-14497080 ] ASF subversion and git services commented on FLUME-2670: Commit 275329ca6c5ebd5212879696fcd0d19f691c1b60 in flume's branch refs/heads/trunk from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=275329c ] FLUME-2670. Modify dependencies to not pull hadoop/hive related jars (Johny Rufus via Hari) Modify dependencies to not pull hadoop/hive related jars Key: FLUME-2670 URL: https://issues.apache.org/jira/browse/FLUME-2670 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2670.patch Recent changes to kite-sdk and flume-ng-auth modules, have introduced hadoop and hive dependency jars to be pulled in the flume distribution. Working/testing on a patch, will upload soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2670) Modify dependencies to not pull hadoop/hive related jars
[ https://issues.apache.org/jira/browse/FLUME-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497078#comment-14497078 ] Hari Shreedharan commented on FLUME-2670: - +1. I removed the change made to TestLog.java though. Modify dependencies to not pull hadoop/hive related jars Key: FLUME-2670 URL: https://issues.apache.org/jira/browse/FLUME-2670 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2670.patch Recent changes to kite-sdk and flume-ng-auth modules, have introduced hadoop and hive dependency jars to be pulled in the flume distribution. Working/testing on a patch, will upload soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2673) Remove unused import in TestLog
[ https://issues.apache.org/jira/browse/FLUME-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497118#comment-14497118 ] ASF subversion and git services commented on FLUME-2673: Commit 364752a43f8ad7f85fd232351a179db017cd2265 in flume's branch refs/heads/trunk from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=364752a ] FLUME-2673. Remove unused import in TestLog (Johny Rufus via Hari) Remove unused import in TestLog Key: FLUME-2673 URL: https://issues.apache.org/jira/browse/FLUME-2673 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2673.patch There is an unused import in TestLog, that depends on javax.ws.rs, which causes compilation to fail when using mvn install -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Branching for 1.6 release
How about switching to hbase-1 as the default build? If we can’t support both, we should still switch to hadoop-2 as default. Thanks, Hari On Wed, Apr 15, 2015 at 3:58 PM, Johny Rufus jru...@cloudera.com wrote: Hi All, 1. We are about to branch for 1.6 release, so would ask the committers to hold off any more commits, until branching is done. 2. Can the committers help with updating Trunk + creating the new branch, then I can proceed with RC related work. Thanks, Rufus
[jira] [Updated] (FLUME-2544) Windows: Incorrect Path Separator used in HDFS path (HDFS Sink)
[ https://issues.apache.org/jira/browse/FLUME-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated FLUME-2544: --- Attachment: FLUME-2544.v2.patch [~hshreedharan] ok looks like that works with both 1.x and 2.x hadoop. Uploading v2 patch with the suggested fix. Windows: Incorrect Path Separator used in HDFS path (HDFS Sink) --- Key: FLUME-2544 URL: https://issues.apache.org/jira/browse/FLUME-2544 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.5.0.1 Reporter: Roshan Naik Assignee: Roshan Naik Labels: windows Attachments: FLUME-2544.patch, FLUME-2544.v2.patch Need to use / for HDFS paths as separator instead of system specific System.getProperty(file.separator) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2673) Remove unused import in TestLog
Johny Rufus created FLUME-2673: -- Summary: Remove unused import in TestLog Key: FLUME-2673 URL: https://issues.apache.org/jira/browse/FLUME-2673 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 There is an unused import in TestLog, that depends on javax.ws.rs, which causes compilation to fail when using mvn install -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2673) Remove unused import in TestLog
[ https://issues.apache.org/jira/browse/FLUME-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497119#comment-14497119 ] ASF subversion and git services commented on FLUME-2673: Commit 4451eddf7e638bf5e8bcd9b91358681fa86aca6e in flume's branch refs/heads/flume-1.6 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4451edd ] FLUME-2673. Remove unused import in TestLog (Johny Rufus via Hari) Remove unused import in TestLog Key: FLUME-2673 URL: https://issues.apache.org/jira/browse/FLUME-2673 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2673.patch There is an unused import in TestLog, that depends on javax.ws.rs, which causes compilation to fail when using mvn install -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2670) Modify dependencies to not pull hadoop/hive related jars
[ https://issues.apache.org/jira/browse/FLUME-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497142#comment-14497142 ] Hudson commented on FLUME-2670: --- FAILURE: Integrated in Flume-trunk-hbase-98 #91 (See [https://builds.apache.org/job/Flume-trunk-hbase-98/91/]) FLUME-2670. Modify dependencies to not pull hadoop/hive related jars (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.gita=commith=275329ca6c5ebd5212879696fcd0d19f691c1b60) * flume-ng-sinks/flume-dataset-sink/pom.xml * flume-ng-auth/pom.xml Modify dependencies to not pull hadoop/hive related jars Key: FLUME-2670 URL: https://issues.apache.org/jira/browse/FLUME-2670 Project: Flume Issue Type: Bug Affects Versions: 1.6 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: 1.6 Attachments: FLUME-2670.patch Recent changes to kite-sdk and flume-ng-auth modules, have introduced hadoop and hive dependency jars to be pulled in the flume distribution. Working/testing on a patch, will upload soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Branching for 1.6 release
Hi All, 1. We are about to branch for 1.6 release, so would ask the committers to hold off any more commits, until branching is done. 2. Can the committers help with updating Trunk + creating the new branch, then I can proceed with RC related work. Thanks, Rufus
[jira] [Resolved] (FLUME-2676) Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch
[ https://issues.apache.org/jira/browse/FLUME-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan resolved FLUME-2676. - Resolution: Fixed Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch --- Key: FLUME-2676 URL: https://issues.apache.org/jira/browse/FLUME-2676 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan Assignee: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2676) Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch
[ https://issues.apache.org/jira/browse/FLUME-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan reassigned FLUME-2676: --- Assignee: Hari Shreedharan Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch --- Key: FLUME-2676 URL: https://issues.apache.org/jira/browse/FLUME-2676 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan Assignee: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2677) Update versions in 1.6.0 branch
Hari Shreedharan created FLUME-2677: --- Summary: Update versions in 1.6.0 branch Key: FLUME-2677 URL: https://issues.apache.org/jira/browse/FLUME-2677 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2652) Documented transaction handling semantics incorrect
[ https://issues.apache.org/jira/browse/FLUME-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497601#comment-14497601 ] ASF subversion and git services commented on FLUME-2652: Commit 15d71c311ed95946195c09b5b2034d9d362c24a0 in flume's branch refs/heads/flume-1.7 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=15d71c3 ] FLUME-2652. Documented transaction handling semantics incorrect in developer guide. (Graham Gear via Hari) Documented transaction handling semantics incorrect --- Key: FLUME-2652 URL: https://issues.apache.org/jira/browse/FLUME-2652 Project: Flume Issue Type: Bug Components: Docs Affects Versions: v1.5.1 Reporter: Graham Gear Priority: Minor Attachments: FLUME-2652-1.patch The Developer Guide Transaction and Source descriptions and Source example are misleading, stating that transactions must be managed within the Source space, when in fact they are managed within the ChannelProcessor. The Source example code does not compile against the current API, nor is the intention Source space transaction management possible with the current ChannelProcessor.processEvent() and BasicTransactionSemantics.begin() implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2652) Documented transaction handling semantics incorrect
[ https://issues.apache.org/jira/browse/FLUME-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497599#comment-14497599 ] Hari Shreedharan commented on FLUME-2652: - +1. Committing to trunk and 1.7 Documented transaction handling semantics incorrect --- Key: FLUME-2652 URL: https://issues.apache.org/jira/browse/FLUME-2652 Project: Flume Issue Type: Bug Components: Docs Affects Versions: v1.5.1 Reporter: Graham Gear Priority: Minor Attachments: FLUME-2652-1.patch The Developer Guide Transaction and Source descriptions and Source example are misleading, stating that transactions must be managed within the Source space, when in fact they are managed within the ChannelProcessor. The Source example code does not compile against the current API, nor is the intention Source space transaction management possible with the current ChannelProcessor.processEvent() and BasicTransactionSemantics.begin() implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2658) Upgrade to Hbase 1.0
[ https://issues.apache.org/jira/browse/FLUME-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497355#comment-14497355 ] Hudson commented on FLUME-2658: --- ABORTED: Integrated in Flume-trunk-hbase-98 #93 (See [https://builds.apache.org/job/Flume-trunk-hbase-98/93/]) FLUME-2658. Upgrade to Hbase 1.0 (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.gita=commith=019ab5207f2d2f0e7b9113b855b37d20dce664bf) * flume-ng-dist/pom.xml * flume-ng-sinks/flume-ng-hbase-sink/pom.xml * flume-ng-sinks/flume-hdfs-sink/pom.xml * flume-ng-sinks/pom.xml * flume-ng-sinks/flume-hive-sink/pom.xml * pom.xml * flume-tools/pom.xml Upgrade to Hbase 1.0 Key: FLUME-2658 URL: https://issues.apache.org/jira/browse/FLUME-2658 Project: Flume Issue Type: Bug Reporter: Roshan Naik Assignee: Roshan Naik Fix For: v1.6.0 Attachments: FLUME-2658-2.patch, FLUME-2658-3.patch, FLUME-2658.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2675) Update pom.xml version in trunk and 1.7 branch
[ https://issues.apache.org/jira/browse/FLUME-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan resolved FLUME-2675. - Resolution: Fixed Updated versions to 1.7-SNAPSHOT Update pom.xml version in trunk and 1.7 branch -- Key: FLUME-2675 URL: https://issues.apache.org/jira/browse/FLUME-2675 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan Assignee: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2675) Update pom.xml version in trunk and 1.7 branch
[ https://issues.apache.org/jira/browse/FLUME-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497567#comment-14497567 ] ASF subversion and git services commented on FLUME-2675: Commit 30dd5cd3e6e8642e5b14e394e0e6c772b3848ae3 in flume's branch refs/heads/flume-1.7 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=30dd5cd ] FLUME-2675. Update pom.xml version in trunk and 1.7 branch Update pom.xml version in trunk and 1.7 branch -- Key: FLUME-2675 URL: https://issues.apache.org/jira/browse/FLUME-2675 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan Assignee: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Jenkins build is back to stable : Flume-trunk-hbase-1 #94
See https://builds.apache.org/job/Flume-trunk-hbase-1/94/
[jira] [Resolved] (FLUME-2677) Update versions in 1.6.0 branch
[ https://issues.apache.org/jira/browse/FLUME-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan resolved FLUME-2677. - Resolution: Fixed Assignee: Hari Shreedharan Done Update versions in 1.6.0 branch --- Key: FLUME-2677 URL: https://issues.apache.org/jira/browse/FLUME-2677 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan Assignee: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2677) Update versions in 1.6.0 branch
[ https://issues.apache.org/jira/browse/FLUME-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497577#comment-14497577 ] ASF subversion and git services commented on FLUME-2677: Commit 8ac759e0694b80b3cb3af74aadd5b6e4dba94f78 in flume's branch refs/heads/flume-1.6 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=8ac759e ] FLUME-2677. Update versions in 1.6 branch Update versions in 1.6.0 branch --- Key: FLUME-2677 URL: https://issues.apache.org/jira/browse/FLUME-2677 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Branching for 1.6 release
Johny, I have created the branches and updated the versions in all branches. Can you please verify the branches look sane? Also, I’d like to make hbase-1 the main release branch, so that needs a patch, that should go into all branches. Thanks, Hari On Wed, Apr 15, 2015 at 4:15 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: How about switching to hbase-1 as the default build? If we can’t support both, we should still switch to hadoop-2 as default. Thanks, Hari On Wed, Apr 15, 2015 at 3:58 PM, Johny Rufus jru...@cloudera.com wrote: Hi All, 1. We are about to branch for 1.6 release, so would ask the committers to hold off any more commits, until branching is done. 2. Can the committers help with updating Trunk + creating the new branch, then I can proceed with RC related work. Thanks, Rufus
[jira] [Commented] (FLUME-2652) Documented transaction handling semantics incorrect
[ https://issues.apache.org/jira/browse/FLUME-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497605#comment-14497605 ] Hari Shreedharan commented on FLUME-2652: - Committed! Thanks Graham! Documented transaction handling semantics incorrect --- Key: FLUME-2652 URL: https://issues.apache.org/jira/browse/FLUME-2652 Project: Flume Issue Type: Bug Components: Docs Affects Versions: v1.5.1 Reporter: Graham Gear Assignee: Graham Gear Priority: Minor Fix For: v1.7.0 Attachments: FLUME-2652-1.patch The Developer Guide Transaction and Source descriptions and Source example are misleading, stating that transactions must be managed within the Source space, when in fact they are managed within the ChannelProcessor. The Source example code does not compile against the current API, nor is the intention Source space transaction management possible with the current ChannelProcessor.processEvent() and BasicTransactionSemantics.begin() implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2671) Add tail file source with the ability of tail file continuously after restart flume.
[ https://issues.apache.org/jira/browse/FLUME-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497539#comment-14497539 ] Otis Gospodnetic commented on FLUME-2671: - Could you have a look at FLUME-2498 and see if that will work for you? The hope is that FLUME-2498 will get into Flume 1.7 next month. Add tail file source with the ability of tail file continuously after restart flume. Key: FLUME-2671 URL: https://issues.apache.org/jira/browse/FLUME-2671 Project: Flume Issue Type: Improvement Components: Sinks+Sources Environment: UNIX-like system Reporter: Frank Yao Priority: Minor Attachments: FLUME-2671-0.patch A Source for tail files. And it will store the offset where source has read in an offset file. This feature enables source to tail continuously after flume restarted. The offset file will rotate when its size over user specified value so that the offset file will not get bigger and bigger until make disk full. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2674) Flume 1.6 release
Hari Shreedharan created FLUME-2674: --- Summary: Flume 1.6 release Key: FLUME-2674 URL: https://issues.apache.org/jira/browse/FLUME-2674 Project: Flume Issue Type: Bug Reporter: Hari Shreedharan Assignee: Johny Rufus Umbrella jira for Flume 1.6 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2675) Update pom.xml version in trunk and 1.7 branch
Hari Shreedharan created FLUME-2675: --- Summary: Update pom.xml version in trunk and 1.7 branch Key: FLUME-2675 URL: https://issues.apache.org/jira/browse/FLUME-2675 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2675) Update pom.xml version in trunk and 1.7 branch
[ https://issues.apache.org/jira/browse/FLUME-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan reassigned FLUME-2675: --- Assignee: Hari Shreedharan Update pom.xml version in trunk and 1.7 branch -- Key: FLUME-2675 URL: https://issues.apache.org/jira/browse/FLUME-2675 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan Assignee: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2676) Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch
[ https://issues.apache.org/jira/browse/FLUME-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497572#comment-14497572 ] ASF subversion and git services commented on FLUME-2676: Commit b92fbdcf0ea34efe47749af872bb68dc98b0cad5 in flume's branch refs/heads/trunk from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=b92fbdc ] FLUME-2676. Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch --- Key: FLUME-2676 URL: https://issues.apache.org/jira/browse/FLUME-2676 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2676) Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch
[ https://issues.apache.org/jira/browse/FLUME-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497573#comment-14497573 ] ASF subversion and git services commented on FLUME-2676: Commit ef88460156e90ac635ec9172e5710bedf8f05a88 in flume's branch refs/heads/flume-1.7 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ef88460 ] FLUME-2676. Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch --- Key: FLUME-2676 URL: https://issues.apache.org/jira/browse/FLUME-2676 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-1796) Upgrade Thrift due to race condition in TThreadSeverPool
[ https://issues.apache.org/jira/browse/FLUME-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497408#comment-14497408 ] Roshan Naik commented on FLUME-1796: [~venkatnrangan] do not change the min hbase version. flume is switching its hbase version to 1.0 Upgrade Thrift due to race condition in TThreadSeverPool Key: FLUME-1796 URL: https://issues.apache.org/jira/browse/FLUME-1796 Project: Flume Issue Type: Bug Components: Test Affects Versions: v1.3.0 Environment: RHEL 5.x based systems, some SUSE versions Reporter: Venkat Ranganathan Assignee: Venkat Ranganathan Priority: Minor Attachments: FLUME-1796.patch While running the Flume unit tests, I observed that under some Linux environments (RHEL 5.x based etc), the Flume thrift source tests were timing out. This is because of a race condition in TThreadPoolServer used by Flume and speicifically in the testlifecycle. I have created a JIRA issue on Thrift regarding this[1]. I have creaed this issue, just so that we cross reference it here. [1] THRIFT-1795 - Race condition in TThreadedServerPool java implementation, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2676) Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch
[ https://issues.apache.org/jira/browse/FLUME-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-2676: Summary: Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch (was: Update flume docs to 1.7-SNAPSHOT in trunk and 1.7 branch) Update flume docs and release notes to 1.7-SNAPSHOT in trunk and 1.7 branch --- Key: FLUME-2676 URL: https://issues.apache.org/jira/browse/FLUME-2676 Project: Flume Issue Type: Sub-task Reporter: Hari Shreedharan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2649) Elasticsearch sink doesn't handle JSON fields correctly
[
https://issues.apache.org/jira/browse/FLUME-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Fiorini updated FLUME-2649:
Attachment: FLUME-2649-4.patch
Addin a new patch [^FLUME-2649-4.patch] because comments sometimes help better
understand the logic :)
Elasticsearch sink doesn't handle JSON fields correctly
---
Key: FLUME-2649
URL: https://issues.apache.org/jira/browse/FLUME-2649
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Reporter: Francis
Assignee: Benjamin Fiorini
Attachments: FLUME-2649-0.patch, FLUME-2649-1.patch,
FLUME-2649-2.patch, FLUME-2649-3.patch, FLUME-2649-4.patch
JSON attributes are treated like normal strings and are escaped by the sink.
For example, if the body or a header contains the following value:
{code:javascript}
{foo:bar}
{code}
It will be added like this in Elasticsearch:
{code:javascript}
{@message: {\foo\:\bar\}}
{code}
We end up with a plain string instead of a valid JSON field.
I think I found how to fix this bug. The source of the problem is caused by
the way a complex field is added. The ES XContent classes are used to parse
the data in the detected format, but then, instead of adding the parsed data,
the string() method is called and it converts it back to a string that is the
same as the initial data! Here is the current code with added comments:
{code}
XContentBuilder tmp = jsonBuilder(); // This tmp builder is completely
useless.
parser = XContentFactory.xContent(contentType).createParser(data);
parser.nextToken();
tmp.copyCurrentStructure(parser); // This copies the whole parsed data in
this tmp builder.
// Here, by calling tmp.string(), we get the parsed data converted back to a
string.
// This means that tmp.string() == String(data)!
// All this parsing for nothing...
// And then, as the field(String, String) method is called on the builder,
and the builder being a jsonBuilder,
// the string will be escaped according to the JSON specifications.
builder.field(fieldName, tmp.string());
{code}
If we really want to take advantage of the XContent classes, we have to add
the parsed data to the builder. To do this, it is as simply as:
{code}
parser = XContentFactory.xContent(contentType).createParser(data);
parser.nextToken();
// Add the field name, but not the value.
builder.field(fieldName);
// This will add the whole parsed content as the value of the field.
builder.copyCurrentStructure(parser);
{code}
I tried this and it works as expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2649) Elasticsearch sink doesn't handle JSON fields correctly
[
https://issues.apache.org/jira/browse/FLUME-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495906#comment-14495906
]
Benjamin Fiorini commented on FLUME-2649:
-
I will post a new review board as soon as the reviews.apache.org website works.
Elasticsearch sink doesn't handle JSON fields correctly
---
Key: FLUME-2649
URL: https://issues.apache.org/jira/browse/FLUME-2649
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Reporter: Francis
Assignee: Benjamin Fiorini
Attachments: FLUME-2649-0.patch, FLUME-2649-1.patch,
FLUME-2649-2.patch, FLUME-2649-3.patch, FLUME-2649-4.patch
JSON attributes are treated like normal strings and are escaped by the sink.
For example, if the body or a header contains the following value:
{code:javascript}
{foo:bar}
{code}
It will be added like this in Elasticsearch:
{code:javascript}
{@message: {\foo\:\bar\}}
{code}
We end up with a plain string instead of a valid JSON field.
I think I found how to fix this bug. The source of the problem is caused by
the way a complex field is added. The ES XContent classes are used to parse
the data in the detected format, but then, instead of adding the parsed data,
the string() method is called and it converts it back to a string that is the
same as the initial data! Here is the current code with added comments:
{code}
XContentBuilder tmp = jsonBuilder(); // This tmp builder is completely
useless.
parser = XContentFactory.xContent(contentType).createParser(data);
parser.nextToken();
tmp.copyCurrentStructure(parser); // This copies the whole parsed data in
this tmp builder.
// Here, by calling tmp.string(), we get the parsed data converted back to a
string.
// This means that tmp.string() == String(data)!
// All this parsing for nothing...
// And then, as the field(String, String) method is called on the builder,
and the builder being a jsonBuilder,
// the string will be escaped according to the JSON specifications.
builder.field(fieldName, tmp.string());
{code}
If we really want to take advantage of the XContent classes, we have to add
the parsed data to the builder. To do this, it is as simply as:
{code}
parser = XContentFactory.xContent(contentType).createParser(data);
parser.nextToken();
// Add the field name, but not the value.
builder.field(fieldName);
// This will add the whole parsed content as the value of the field.
builder.copyCurrentStructure(parser);
{code}
I tried this and it works as expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Review Request 33215: FLUME-2649 ElasticSearch sink should index nested JSON
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33215/ --- Review request for Flume. Repository: flume-git Description --- Elasticsearch sink fails to index malformed nested JSON Diffs - flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java de0acf4 flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java 2f8fd6d flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java 78e1665 Diff: https://reviews.apache.org/r/33215/diff/ Testing --- Thanks, Benjamin Fiorini
