Hi, This is my Flume Configuration File
agent.sources = tail agent.channels = memoryChannel agent.sinks = loggerSink agent.sinks = phoenix-sink agent.sources.tail.type = exec agent.sources.tail.command = tail -f /var/log/httpd/access_log agent.sources.tail.channels = memoryChannel agent.sinks.loggerSink.channel = memoryChannel agent.sinks.loggerSink.type = logger agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 100 agent.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink agent.sinks.phoenix-sink.channel=memoryChannel agent.sinks.phoenix-sink.batchSize=5 agent.sinks.phoenix-sink.table=S1.APACHE agent.sinks.phoenix-sink.zookeeperQuorum=nn01 agent.sinks.phoenix-sink.serializer=REGEX agent.sinks.phoenix-sink.serializer.rowkeyType=uuid agent.sinks.phoenix-sink.ddl=CREATE TABLE IF NOT EXISTS S1.APACHE (uid varchar NOT NULL,host varchar,identity varchar,user varchar,time varchar,method varchar,request varchar,protocol varchar,status INTEGER,size INTEGER,referer varchar,agent varchar,f_host varchar CONSTRAINT pk PRIMARY KEY (uid)) #agent.sinks.phoenix-sink.serializer.regex="([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\]) \"([^ ]+) ([^ ]+) ([^\"]+)\" (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\"[^\"]*\") ([^ \"]$ #agent.sinks.phoenix-sink.serializer.regex="([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\]) \"([^ ]+) ([^ ]+) ([^\"]+)\" (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\"[^\"]*\") ([^ \"]*|\"[^\"]*\"))?" agent.sinks.phoenix-sink.serializer.regex=([^ ]*) ([^ ]*) ([^ ]*) ([^ ]* [^ ]*) "([^\"]+)\" (-|[0-9]*) (-|[0-9]*) "([^ ]*)" "([^\"]+)\" agent.sinks.phoenix-sink.serializer.columns=host,identity,user,time,method,request,protocol,status,size,referer,agent agent.sinks.phoenix-sink.serializer.headers=f_host This Is my Apache log File Structure 127.0.0.1 - - [20/Dec/2014:17:11:06 +0530] "GET / HTTP/1.0" 403 4954 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" 127.0.0.1 - - [20/Dec/2014:17:16:06 +0530] "GET / HTTP/1.0" 403 4954 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" 127.0.0.1 - - [20/Dec/2014:17:21:06 +0530] "GET / HTTP/1.0" 403 4954 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" 127.0.0.1 - - [20/Dec/2014:17:26:06 +0530] "GET / HTTP/1.0" 403 4954 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" 127.0.0.1 - - [20/Dec/2014:17:31:06 +0530] "GET / HTTP/1.0" 403 4954 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" 127.0.0.1 - - [20/Dec/2014:17:36:06 +0530] "GET / HTTP/1.0" 403 4954 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" 127.0.0.1 - - [20/Dec/2014:17:41:06 +0530] "GET / HTTP/1.0" 403 4954 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" 127.0.0.1 - - [20/Dec/2014:17:46:06 +0530] "GET / HTTP/1.0" 403 4954 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" 127.0.0.1 - - [20/Dec/2014:17:51:06 +0530] "GET / HTTP/1.0" 403 4954 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" 127.0.0.1 - - [20/Dec/2014:17:56:06 +0530] "GET / HTTP/1.0" 403 4954 "-" "check_http/v2.0.3 (nagios-plugins 2.0.3)" Iam using phoenix 4.2.1 Hbase 0.98.8 and Sorry i enable DEBUG Mode in flume . it shows only INFO as usual when executing this flume-ng agent -c conf -f /opt/flume/conf/apache.conf -n agent -Dflume.root.looger=DEBUG,console Thanks Divya N On Sat, Dec 20, 2014 at 2:14 AM, Ravi Kiran <[email protected]> wrote: > Hi Divya, > > Also, can you confirm if the regex given in the configuration matches > the access log . To confirm , is it possible to set the logging level to > debug as there is debug log entry if the event doesn't match the regex > given in the configuration. > We have a test case for processing apache logs > https://github.com/apache/phoenix/blob/master/phoenix-flume/src/it/java/org/apache/phoenix/flume/RegexEventSerializerIT.java#testApacheLogRegex > which can help you with the regex > Happy to help!! > > Regards > Ravi > > On Fri, Dec 19, 2014 at 11:19 AM, Ravi Kiran <[email protected]> > wrote: >> >> Hi Nagarajan, >> >> Do you see any exceptions in the logs ? Can you please give it a try >> to ingest > 100 records and see if that works. Also, can you please share >> the version of Phoenix you are using. >> >> Regards >> Ravi >> >> On Thu, Dec 18, 2014 at 10:36 PM, Divya Nagarajan <[email protected] >> > wrote: >>> >>> >>> H i, >>> I tried with 5 as batchsize,still data is not upserted into phoenix. >>> >>> >>> 14/12/19 12:03:51 INFO zookeeper.ClientCnxn: EventThread shut down >>> 14/12/19 12:03:51 INFO zookeeper.ZooKeeper: Session: 0x14a60b4631e003d >>> closed >>> 14/12/19 12:03:55 INFO serializer.BaseEventSerializer: the upsert >>> statement is UPSERT INTO APACHE_LOG4 ("HOST", "IDENTITY", "USER", "TIME", >>> "METHOD", "REQUEST", "PROTOCOL", "STATUS", "SIZE", "REFERER", "AGENT", >>> "F_HOST", "UID") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) >>> 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] >>> events was [0] seconds >>> 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] >>> events was [0] seconds >>> 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] >>> events was [0] seconds >>> 14/12/19 12:03:58 INFO sink.PhoenixSink: Time taken to process [1] >>> events was [3] seconds >>> 14/12/19 12:04:02 INFO sink.PhoenixSink: Time taken to process [0] >>> events was [3] seconds >>> 14/12/19 12 >>> >>> >>> Thanks >>> >>> Divya N >>> >>> >>> On Fri, Dec 19, 2014 at 6:10 AM, Ravi Kiran <[email protected]> >>> wrote: >>>> >>>> >>>> Hi Nagarajan, >>>> >>>> Apparently, we do batches of 100 by default for each commit . You >>>> can decrease that number if you would like to. >>>> http://phoenix.apache.org/flume.html >>>> >>>> Regards >>>> Ravi >>>> >>>> Hi, >>>>> >>>>> As we are working in flume to store apache log into hbase/phoenix >>>>> using phoenix-flume jars.Using flume conf file we could create table >>>>> in phoenix and see the events get processed.while querying through >>>>> phoenix there is no record stored.please suggest us steps to process >>>>> apache access log as source and store data in phoenix as sink. >>>>> >>>>> Hbase : 0.98.8 >>>>> Phoenix:4.2.1 >>>>> >>>>> Flume Configuration: >>>>> =============== >>>>> >>>>> agent.channels.memory-channel.type = memory >>>>> agent.sources.tail-source.type = exec >>>>> agent.sources.tail-source.command = tail -F /var/log/httpd/access_log >>>>> agent.sources.tail-source.channels = memory-channel >>>>> >>>>> agent.channels.memory-channel.type=memory >>>>> agent.channels.memory-channel.transactionCapacity=100 >>>>> agent.channels.memory-channel.byteCapacityBufferPercentage=20 >>>>> >>>>> >>>>> agent.channels = memory-channel >>>>> agent.sources = tail-source >>>>> agent.sinks = phoenix-sink >>>>> >>>>> >>>>> agent.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink >>>>> agent.sinks.phoenix-sink.channel=memory-channel >>>>> agent.sinks.phoenix-sink.batchSize=100 >>>>> agent.sinks.phoenix-sink.table=APACHE_LOGS >>>>> agent.sinks.phoenix-sink.ddl=CREATE TABLE IF NOT EXISTS >>>>> APACHE_LOGS(uid VARCHAR NOT NULL,host VARCHAR,identity VARCHAR,user >>>>> VARCHAR,time VARCHAR,method VARCHAR,request VARCHAR,protocol >>>>> VARCHAR,status INTEGER,size INTEGER,referer VARCHAR,agent >>>>> VARCHAR,f_host VARCHAR CONSTRAINT pk PRIMARY KEY(uid)) >>>>> agent.sinks.phoenix-sink.zookeeperQuorum=dn02 >>>>> agent.sinks.phoenix-sink.serializer=REGEX >>>>> agent.sinks.phoenix-sink.serializer.regex="^([\\d.]+) (\\S+) (\\S+) >>>>> \\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(.+?)\" (\\d{3}) (\\d+) \"([^\"]+)\" >>>>> \"([^\"]+)\"" >>>>> agent.sinks.phoenix-sink.serializer.rowkeyType=uuid >>>>> >>>>> agent.sinks.phoenix-sink.serializer.columns=host,identity,user,time,method,request,protocol,status,size,referer,agent >>>>> agent.sinks.phoenix-sink.serializer.headers=f_host >>>>> >>>>> flume-ng agent -c conf -f /opt/flume/conf/phoenix.conf -n agent >>>>> >>>>> * >>>>> * >>>>> * >>>>> * >>>>> * >>>>> * >>>>> >>>>> 14/12/18 12:53:09 INFO >>>>> client.HConnectionManager$HConnectionImplementation: Closing master >>>>> protocol: MasterService >>>>> 14/12/18 12:53:09 INFO >>>>> client.HConnectionManager$HConnectionImplementation: Closing zookeeper >>>>> sessionid=0x14a5b888d18003e >>>>> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003e >>>>> closed >>>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down >>>>> 14/12/18 12:53:09 INFO query.ConnectionQueryServicesImpl: Found >>>>> quorum: nn01:2181 >>>>> 14/12/18 12:53:09 INFO zookeeper.RecoverableZooKeeper: Process >>>>> identifier=hconnection-0x843735 connecting to ZooKeeper >>>>> ensemble=nn01:2181 >>>>> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Initiating client >>>>> connection, connectString=nn01:2181 sessionTimeout=180000 >>>>> watcher=hconnection-0x843735, quorum=nn01:2181, baseZNode=/hbase >>>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Opening socket connection >>>>> to server nn01/10.10.10.25:2181. Will not attempt to authenticate >>>>> using SASL (unknown error) >>>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Socket connection >>>>> established to nn01/10.10.10.25:2181, initiating session >>>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Session establishment >>>>> complete on server nn01/10.10.10.25:2181, sessionid = >>>>> 0x14a5b888d18003f, negotiated timeout = 40000 >>>>> 14/12/18 12:53:09 INFO >>>>> client.HConnectionManager$HConnectionImplementation: Closing master >>>>> protocol: MasterService >>>>> 14/12/18 12:53:09 INFO >>>>> client.HConnectionManager$HConnectionImplementation: Closing zookeeper >>>>> sessionid=0x14a5b888d18003f >>>>> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003f >>>>> closed >>>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down >>>>> 14/12/18 12:53:15 INFO serializer.BaseEventSerializer: the upsert >>>>> statement is UPSERT INTO APACHE_LOG4 ("HOST", "IDENTITY", "USER", >>>>> "TIME", "METHOD", "REQUEST", "PROTOCOL", "STATUS", "SIZE", "REFERER", >>>>> "AGENT", "F_HOST", "UID") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>>>> ?) >>>>> 14/12/18 12:53:18 INFO sink.PhoenixSink: Time taken to process [10] >>>>> events was [3] seconds >>>>> 14/12/18 12:53:22 INFO sink.PhoenixSink: Time taken to process [0] >>>>> events was [3] seconds >>>>> 14/12/18 12:53:27 INFO sink.PhoenixSink: Time taken to process [0] >>>>> events was [3] seconds >>>>> 14/12/18 12:53:33 INFO sink.PhoenixSink: Time taken to process [0] >>>>> events was [3] seconds >>>>> 14/12/18 12:53:40 INFO sink.PhoenixSink: Time taken to process [0] >>>>> events was [3] seconds >>>>> 14/12/18 12:53:48 INFO sink.PhoenixSink: Time taken to process [0] >>>>> events was [3] seconds >>>>> 14/12/18 12:53:56 INFO sink.PhoenixSink: Time taken to process [0] >>>>> events was [3] seconds >>>>> 14/12/18 12:54:04 INFO sink.PhoenixSink: Time taken to process [0] >>>>> events was [3] seconds >>>>> 14/12/18 12:54:12 INFO sink.PhoenixSink: Time taken to process [0] >>>>> events was [3] seconds >>>>> 14/12/18 12:54:20 INFO sink.PhoenixSink: Time taken to process [0] >>>>> events was [3] seconds >>>>> 14/12/18 12:54:28 INFO sink.PhoenixSink: Time taken to process [1] >>>>> events was [3] seconds >>>>> 14/12/18 12:54:36 INFO sink.PhoenixSink: Time taken to process [0] >>>>> events was [3] seconds >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Divya N >>>>> >>>>
