[jira] [Updated] (FLUME-2802) Folder name interceptor

2015-09-26 Thread Eran W (JIRA)

 [ 
https://issues.apache.org/jira/browse/FLUME-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eran W updated FLUME-2802:
--
Attachment: (was: flume-2802.patch)

> Folder name interceptor
> ---
>
> Key: FLUME-2802
> URL: https://issues.apache.org/jira/browse/FLUME-2802
> Project: Flume
>  Issue Type: New Feature
>Reporter: Eran W
> Attachments: flume-2802.patch
>
>
> This interceptor retrieve the last folder name from the 
> SpoolDir.fileHeaderKey and set it to the given folderKey.
> This is allow users to set the target hdfs directory based on the source 
> directory and not the whole path or file name. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (FLUME-2802) Folder name interceptor

2015-09-26 Thread Eran W (JIRA)

 [ 
https://issues.apache.org/jira/browse/FLUME-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eran W updated FLUME-2802:
--
Attachment: flume-2802.patch

interceptor code

> Folder name interceptor
> ---
>
> Key: FLUME-2802
> URL: https://issues.apache.org/jira/browse/FLUME-2802
> Project: Flume
>  Issue Type: New Feature
>Reporter: Eran W
> Attachments: flume-2802.patch, flume-2802.patch
>
>
> This interceptor retrieve the last folder name from the 
> SpoolDir.fileHeaderKey and set it to the given folderKey.
> This is allow users to set the target hdfs directory based on the source 
> directory and not the whole path or file name. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Issue Comment Deleted] (FLUME-2802) Folder name interceptor

2015-09-26 Thread Eran W (JIRA)

 [ 
https://issues.apache.org/jira/browse/FLUME-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eran W updated FLUME-2802:
--
Comment: was deleted

(was: Initial commit for the interceptor)

> Folder name interceptor
> ---
>
> Key: FLUME-2802
> URL: https://issues.apache.org/jira/browse/FLUME-2802
> Project: Flume
>  Issue Type: New Feature
>Reporter: Eran W
> Attachments: flume-2802.patch
>
>
> This interceptor retrieve the last folder name from the 
> SpoolDir.fileHeaderKey and set it to the given folderKey.
> This is allow users to set the target hdfs directory based on the source 
> directory and not the whole path or file name. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (FLUME-2802) Folder name interceptor

2015-09-26 Thread Eran W (JIRA)

 [ 
https://issues.apache.org/jira/browse/FLUME-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eran W updated FLUME-2802:
--
Attachment: (was: flume-2802.patch)

> Folder name interceptor
> ---
>
> Key: FLUME-2802
> URL: https://issues.apache.org/jira/browse/FLUME-2802
> Project: Flume
>  Issue Type: New Feature
>Reporter: Eran W
> Attachments: FLUME-2802.patch
>
>
> This interceptor retrieve the last folder name from the 
> SpoolDir.fileHeaderKey and set it to the given folderKey.
> This is allow users to set the target hdfs directory based on the source 
> directory and not the whole path or file name. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (FLUME-2802) Folder name interceptor

2015-09-26 Thread Eran W (JIRA)

 [ 
https://issues.apache.org/jira/browse/FLUME-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eran W updated FLUME-2802:
--
Attachment: FLUME-2802.patch

new patch integrated with flume source.

> Folder name interceptor
> ---
>
> Key: FLUME-2802
> URL: https://issues.apache.org/jira/browse/FLUME-2802
> Project: Flume
>  Issue Type: New Feature
>Reporter: Eran W
> Attachments: FLUME-2802.patch
>
>
> This interceptor retrieve the last folder name from the 
> SpoolDir.fileHeaderKey and set it to the given folderKey.
> This is allow users to set the target hdfs directory based on the source 
> directory and not the whole path or file name. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


contributing to flume

2015-09-26 Thread IT CTO
Hi,
I am new to flume and want to be able to contribute to flume.
I opened a jira issue (flume-2802) but I can't assign it to me so I can't
change the status to get the patch reviewed.
Can someone help here?

Eran
-- 
Eran | "You don't need eyes to see, you need vision" (Faithless)


[jira] [Created] (FLUME-2803) File channel replay for long time

2015-09-26 Thread yinghua_zh (JIRA)
yinghua_zh created FLUME-2803:
-

 Summary: File channel replay for long time
 Key: FLUME-2803
 URL: https://issues.apache.org/jira/browse/FLUME-2803
 Project: Flume
  Issue Type: Question
Reporter: yinghua_zh


When the agent restart (the host restarted),and the checkpoint data is 
destroyed,the flume agent replay for two hours long. I found that the log 
replay method "processCommit" invoke for a long time? How can handle it and it 
can replay quickly when the agent run normally and it happened? and in what 
case ,it will be happened again?thanks




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (FLUME-2804) Hive sink - abort remaining transactions on shutdown

2015-09-26 Thread Sriharsha Chintalapani (JIRA)

 [ 
https://issues.apache.org/jira/browse/FLUME-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriharsha Chintalapani updated FLUME-2804:
--
Attachment: FLUME-2804.patch

> Hive sink - abort remaining transactions on shutdown
> 
>
> Key: FLUME-2804
> URL: https://issues.apache.org/jira/browse/FLUME-2804
> Project: Flume
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
> Attachments: FLUME-2804.patch
>
>
> Currently the hive sink does not explicitly abort unused transactions. 
> Although these eventually timeout on the hive side, it is preferable to 
> explicitly abort them so that the associated locks on the hive 
> table/partition are released. As long as the locks stay open, the 
> table/partition cannot be dropped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: contributing to flume

2015-09-26 Thread Arvind Prabhakar
Hi Eran,

I added you to the contributors list and assigned the Jira to you. You
should be able to make the status changes yourself.

Regards,
Arvind Prabhakar

On Sat, Sep 26, 2015 at 4:20 AM, IT CTO  wrote:

> Hi,
> I am new to flume and want to be able to contribute to flume.
> I opened a jira issue (flume-2802) but I can't assign it to me so I can't
> change the status to get the patch reviewed.
> Can someone help here?
>
> Eran
> --
> Eran | "You don't need eyes to see, you need vision" (Faithless)
>


[jira] [Assigned] (FLUME-2802) Folder name interceptor

2015-09-26 Thread Arvind Prabhakar (JIRA)

 [ 
https://issues.apache.org/jira/browse/FLUME-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arvind Prabhakar reassigned FLUME-2802:
---

Assignee: Eran W

> Folder name interceptor
> ---
>
> Key: FLUME-2802
> URL: https://issues.apache.org/jira/browse/FLUME-2802
> Project: Flume
>  Issue Type: New Feature
>Reporter: Eran W
>Assignee: Eran W
> Attachments: FLUME-2802.patch
>
>
> This interceptor retrieve the last folder name from the 
> SpoolDir.fileHeaderKey and set it to the given folderKey.
> This is allow users to set the target hdfs directory based on the source 
> directory and not the whole path or file name. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (FLUME-2804) ive sink - abort remaining transactions on shutdown

2015-09-26 Thread Sriharsha Chintalapani (JIRA)
Sriharsha Chintalapani created FLUME-2804:
-

 Summary: ive sink - abort remaining transactions on shutdown
 Key: FLUME-2804
 URL: https://issues.apache.org/jira/browse/FLUME-2804
 Project: Flume
  Issue Type: Bug
Reporter: Sriharsha Chintalapani


Currently the hive sink does not explicitly abort unused transactions. Although 
these eventually timeout on the hive side, it is preferable to explicitly abort 
them so that the associated locks on the hive table/partition are released. As 
long as the locks stay open, the table/partition cannot be dropped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (FLUME-2804) Hive sink - abort remaining transactions on shutdown

2015-09-26 Thread Sriharsha Chintalapani (JIRA)

 [ 
https://issues.apache.org/jira/browse/FLUME-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriharsha Chintalapani updated FLUME-2804:
--
Summary: Hive sink - abort remaining transactions on shutdown  (was: ive 
sink - abort remaining transactions on shutdown)

> Hive sink - abort remaining transactions on shutdown
> 
>
> Key: FLUME-2804
> URL: https://issues.apache.org/jira/browse/FLUME-2804
> Project: Flume
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>
> Currently the hive sink does not explicitly abort unused transactions. 
> Although these eventually timeout on the hive side, it is preferable to 
> explicitly abort them so that the associated locks on the hive 
> table/partition are released. As long as the locks stay open, the 
> table/partition cannot be dropped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)