[
https://issues.apache.org/jira/browse/FLUME-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262311#comment-13262311
]
Mubarak Seyed edited comment on FLUME-1150 at 4/26/12 1:55 AM:
---------------------------------------------------------------
If we dont change the
{code}
TAG_REGEX = "\\%(\\w|\\%)|\\%\\{([\\w\\.-]+)\\}";
{code}
then we can specify the pattern as
sinkName.hdfs.path = /logs/%{tableName}/%Y%m%d/%H:%M:%S
or
sinkName.hdfs.path = /logs/%{tableName}
and header should have key as "tableName" and its value (for example:
headers.add("tableName","table1");
if there is no key for "tableName" or value is null then we can fallback to
"UNKNOWN" (so the path would be /logs/UNKNOWN
if i specify pattern as /logs/%{tableName|UNKNOWN} then we need to change the
TAX_REGEX
I tested without specifying "|UNKNOWN" part, it works fine without any code
change. Thanks.
was (Author: mubarakseyed):
If we dont change the
{code}
TAG_REGEX = "\\%(\\w|\\%)|\\%\\{([\\w\\.-]+)\\}";
{code}
then we can specify the pattern as
{code}
sinkName.hdfs.path = /logs/%{tableName}/%Y%m%d/%H:%M:%S
or
sinkName.hdfs.path = /logs/%{tableName}
and header should have key as "tableName" and its value (for example:
headers.add("tableName","table1");
if there is no key for "tableName" or value is null then we can fallback to
"UNKNOWN" (so the path would be /logs/UNKNOWN
if i specify pattern as /logs/%{tableName|UNKNOWN} then we need to change the
TAX_REGEX
{code}
I tested without specifying "|UNKNOWN" part, it works fine without any code
change. Thanks.
> Need a way to specify bucket in HDFSEventSink from client
> ---------------------------------------------------------
>
> Key: FLUME-1150
> URL: https://issues.apache.org/jira/browse/FLUME-1150
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Reporter: Mubarak Seyed
>
> We need a way to specify the bucket when event is streamed from client.
> For instance, if i have 5 different log categories (like category [1-5]), if
> i use single source->channel->hdfsSink then all the logs goes to one
> specified directory in HDFS (/logs/<destination>) but we need something like
> /logs/category1
> /category2
> /category3
> /category4
> /category5
> I can use multiplexing (1 source, N channel, and N sinks) but i need to use a
> multiplexing channel processor to route the events(categories) to the proper
> sink.
> What i am thinking is if i specify the category name in header in Event,
> HDFSEventSink (and its helper classes) can create a bucket based on header
> value (and then escape-sequence needs to be honored).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
