Hi Divya, Also, can you confirm if the regex given in the configuration matches the access log . To confirm , is it possible to set the logging level to debug as there is debug log entry if the event doesn't match the regex given in the configuration. We have a test case for processing apache logs https://github.com/apache/phoenix/blob/master/phoenix-flume/src/it/java/org/apache/phoenix/flume/RegexEventSerializerIT.java#testApacheLogRegex which can help you with the regex Happy to help!!
Regards Ravi On Fri, Dec 19, 2014 at 11:19 AM, Ravi Kiran <[email protected]> wrote: > > Hi Nagarajan, > > Do you see any exceptions in the logs ? Can you please give it a try > to ingest > 100 records and see if that works. Also, can you please share > the version of Phoenix you are using. > > Regards > Ravi > > On Thu, Dec 18, 2014 at 10:36 PM, Divya Nagarajan <[email protected]> > wrote: >> >> >> H i, >> I tried with 5 as batchsize,still data is not upserted into phoenix. >> >> >> 14/12/19 12:03:51 INFO zookeeper.ClientCnxn: EventThread shut down >> 14/12/19 12:03:51 INFO zookeeper.ZooKeeper: Session: 0x14a60b4631e003d >> closed >> 14/12/19 12:03:55 INFO serializer.BaseEventSerializer: the upsert >> statement is UPSERT INTO APACHE_LOG4 ("HOST", "IDENTITY", "USER", "TIME", >> "METHOD", "REQUEST", "PROTOCOL", "STATUS", "SIZE", "REFERER", "AGENT", >> "F_HOST", "UID") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) >> 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] events >> was [0] seconds >> 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] events >> was [0] seconds >> 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] events >> was [0] seconds >> 14/12/19 12:03:58 INFO sink.PhoenixSink: Time taken to process [1] events >> was [3] seconds >> 14/12/19 12:04:02 INFO sink.PhoenixSink: Time taken to process [0] events >> was [3] seconds >> 14/12/19 12 >> >> >> Thanks >> >> Divya N >> >> >> On Fri, Dec 19, 2014 at 6:10 AM, Ravi Kiran <[email protected]> >> wrote: >>> >>> >>> Hi Nagarajan, >>> >>> Apparently, we do batches of 100 by default for each commit . You >>> can decrease that number if you would like to. >>> http://phoenix.apache.org/flume.html >>> >>> Regards >>> Ravi >>> >>> Hi, >>>> >>>> As we are working in flume to store apache log into hbase/phoenix >>>> using phoenix-flume jars.Using flume conf file we could create table >>>> in phoenix and see the events get processed.while querying through >>>> phoenix there is no record stored.please suggest us steps to process >>>> apache access log as source and store data in phoenix as sink. >>>> >>>> Hbase : 0.98.8 >>>> Phoenix:4.2.1 >>>> >>>> Flume Configuration: >>>> =============== >>>> >>>> agent.channels.memory-channel.type = memory >>>> agent.sources.tail-source.type = exec >>>> agent.sources.tail-source.command = tail -F /var/log/httpd/access_log >>>> agent.sources.tail-source.channels = memory-channel >>>> >>>> agent.channels.memory-channel.type=memory >>>> agent.channels.memory-channel.transactionCapacity=100 >>>> agent.channels.memory-channel.byteCapacityBufferPercentage=20 >>>> >>>> >>>> agent.channels = memory-channel >>>> agent.sources = tail-source >>>> agent.sinks = phoenix-sink >>>> >>>> >>>> agent.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink >>>> agent.sinks.phoenix-sink.channel=memory-channel >>>> agent.sinks.phoenix-sink.batchSize=100 >>>> agent.sinks.phoenix-sink.table=APACHE_LOGS >>>> agent.sinks.phoenix-sink.ddl=CREATE TABLE IF NOT EXISTS >>>> APACHE_LOGS(uid VARCHAR NOT NULL,host VARCHAR,identity VARCHAR,user >>>> VARCHAR,time VARCHAR,method VARCHAR,request VARCHAR,protocol >>>> VARCHAR,status INTEGER,size INTEGER,referer VARCHAR,agent >>>> VARCHAR,f_host VARCHAR CONSTRAINT pk PRIMARY KEY(uid)) >>>> agent.sinks.phoenix-sink.zookeeperQuorum=dn02 >>>> agent.sinks.phoenix-sink.serializer=REGEX >>>> agent.sinks.phoenix-sink.serializer.regex="^([\\d.]+) (\\S+) (\\S+) >>>> \\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(.+?)\" (\\d{3}) (\\d+) \"([^\"]+)\" >>>> \"([^\"]+)\"" >>>> agent.sinks.phoenix-sink.serializer.rowkeyType=uuid >>>> >>>> agent.sinks.phoenix-sink.serializer.columns=host,identity,user,time,method,request,protocol,status,size,referer,agent >>>> agent.sinks.phoenix-sink.serializer.headers=f_host >>>> >>>> flume-ng agent -c conf -f /opt/flume/conf/phoenix.conf -n agent >>>> >>>> * >>>> * >>>> * >>>> * >>>> * >>>> * >>>> >>>> 14/12/18 12:53:09 INFO >>>> client.HConnectionManager$HConnectionImplementation: Closing master >>>> protocol: MasterService >>>> 14/12/18 12:53:09 INFO >>>> client.HConnectionManager$HConnectionImplementation: Closing zookeeper >>>> sessionid=0x14a5b888d18003e >>>> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003e >>>> closed >>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down >>>> 14/12/18 12:53:09 INFO query.ConnectionQueryServicesImpl: Found >>>> quorum: nn01:2181 >>>> 14/12/18 12:53:09 INFO zookeeper.RecoverableZooKeeper: Process >>>> identifier=hconnection-0x843735 connecting to ZooKeeper >>>> ensemble=nn01:2181 >>>> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Initiating client >>>> connection, connectString=nn01:2181 sessionTimeout=180000 >>>> watcher=hconnection-0x843735, quorum=nn01:2181, baseZNode=/hbase >>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Opening socket connection >>>> to server nn01/10.10.10.25:2181. Will not attempt to authenticate >>>> using SASL (unknown error) >>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Socket connection >>>> established to nn01/10.10.10.25:2181, initiating session >>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Session establishment >>>> complete on server nn01/10.10.10.25:2181, sessionid = >>>> 0x14a5b888d18003f, negotiated timeout = 40000 >>>> 14/12/18 12:53:09 INFO >>>> client.HConnectionManager$HConnectionImplementation: Closing master >>>> protocol: MasterService >>>> 14/12/18 12:53:09 INFO >>>> client.HConnectionManager$HConnectionImplementation: Closing zookeeper >>>> sessionid=0x14a5b888d18003f >>>> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003f >>>> closed >>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down >>>> 14/12/18 12:53:15 INFO serializer.BaseEventSerializer: the upsert >>>> statement is UPSERT INTO APACHE_LOG4 ("HOST", "IDENTITY", "USER", >>>> "TIME", "METHOD", "REQUEST", "PROTOCOL", "STATUS", "SIZE", "REFERER", >>>> "AGENT", "F_HOST", "UID") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>>> ?) >>>> 14/12/18 12:53:18 INFO sink.PhoenixSink: Time taken to process [10] >>>> events was [3] seconds >>>> 14/12/18 12:53:22 INFO sink.PhoenixSink: Time taken to process [0] >>>> events was [3] seconds >>>> 14/12/18 12:53:27 INFO sink.PhoenixSink: Time taken to process [0] >>>> events was [3] seconds >>>> 14/12/18 12:53:33 INFO sink.PhoenixSink: Time taken to process [0] >>>> events was [3] seconds >>>> 14/12/18 12:53:40 INFO sink.PhoenixSink: Time taken to process [0] >>>> events was [3] seconds >>>> 14/12/18 12:53:48 INFO sink.PhoenixSink: Time taken to process [0] >>>> events was [3] seconds >>>> 14/12/18 12:53:56 INFO sink.PhoenixSink: Time taken to process [0] >>>> events was [3] seconds >>>> 14/12/18 12:54:04 INFO sink.PhoenixSink: Time taken to process [0] >>>> events was [3] seconds >>>> 14/12/18 12:54:12 INFO sink.PhoenixSink: Time taken to process [0] >>>> events was [3] seconds >>>> 14/12/18 12:54:20 INFO sink.PhoenixSink: Time taken to process [0] >>>> events was [3] seconds >>>> 14/12/18 12:54:28 INFO sink.PhoenixSink: Time taken to process [1] >>>> events was [3] seconds >>>> 14/12/18 12:54:36 INFO sink.PhoenixSink: Time taken to process [0] >>>> events was [3] seconds >>>> >>>> >>>> Thanks >>>> >>>> Divya N >>>> >>>
