Hi,
As we are working in flume to store apache log into hbase/phoenix using
phoenix-flume jars.Using flume conf file we could create table in phoenix
and see the events get processed.while querying through phoenix there is
no record stored.please suggest us steps to process apache access log as
source and store data in phoenix as sink.
Hbase : 0.98.8
Phoenix:4.2.1
Flume Configuration:
===============
agent.channels.memory-channel.type = memory
agent.sources.tail-source.type = exec
agent.sources.tail-source.command = tail -F /var/log/httpd/access_log
agent.sources.tail-source.channels = memory-channel
agent.channels.memory-channel.type=memory
agent.channels.memory-channel.transactionCapacity=100
agent.channels.memory-channel.byteCapacityBufferPercentage=20
agent.channels = memory-channel
agent.sources = tail-source
agent.sinks = phoenix-sink
agent.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink
agent.sinks.phoenix-sink.channel=memory-channel
agent.sinks.phoenix-sink.batchSize=100
agent.sinks.phoenix-sink.table=APACHE_LOGS
agent.sinks.phoenix-sink.ddl=CREATE TABLE IF NOT EXISTS APACHE_LOGS(uid
VARCHAR NOT NULL,host VARCHAR,identity VARCHAR,user VARCHAR,time
VARCHAR,method VARCHAR,request VARCHAR,protocol VARCHAR,status INTEGER,size
INTEGER,referer VARCHAR,agent VARCHAR,f_host VARCHAR CONSTRAINT pk PRIMARY
KEY(uid))
agent.sinks.phoenix-sink.zookeeperQuorum=dn02
agent.sinks.phoenix-sink.serializer=REGEX
agent.sinks.phoenix-sink.serializer.regex="^([\\d.]+) (\\S+) (\\S+)
\\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(.+?)\" (\\d{3}) (\\d+) \"([^\"]+)\"
\"([^\"]+)\""
agent.sinks.phoenix-sink.serializer.rowkeyType=uuid
agent.sinks.phoenix-sink.serializer.columns=host,identity,user,time,method,request,protocol,status,size,referer,agent
agent.sinks.phoenix-sink.serializer.headers=f_host
flume-ng agent -c conf -f /opt/flume/conf/phoenix.conf -n agent
*
*
*
*
*
*
14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation:
Closing master protocol: MasterService
14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x14a5b888d18003e
14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003e
closed
14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/18 12:53:09 INFO query.ConnectionQueryServicesImpl: Found quorum:
nn01:2181
14/12/18 12:53:09 INFO zookeeper.RecoverableZooKeeper: Process
identifier=hconnection-0x843735 connecting to ZooKeeper ensemble=nn01:2181
14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=nn01:2181 sessionTimeout=180000 watcher=hconnection-0x843735,
quorum=nn01:2181, baseZNode=/hbase
14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Opening socket connection to
server nn01/10.10.10.25:2181. Will not attempt to authenticate using SASL
(unknown error)
14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Socket connection established
to nn01/10.10.10.25:2181, initiating session
14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Session establishment complete
on server nn01/10.10.10.25:2181, sessionid = 0x14a5b888d18003f, negotiated
timeout = 40000
14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation:
Closing master protocol: MasterService
14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x14a5b888d18003f
14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003f
closed
14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/18 12:53:15 INFO serializer.BaseEventSerializer: the upsert
statement is UPSERT INTO APACHE_LOG4 ("HOST", "IDENTITY", "USER", "TIME",
"METHOD", "REQUEST", "PROTOCOL", "STATUS", "SIZE", "REFERER", "AGENT",
"F_HOST", "UID") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
14/12/18 12:53:18 INFO sink.PhoenixSink: Time taken to process [10] events
was [3] seconds
14/12/18 12:53:22 INFO sink.PhoenixSink: Time taken to process [0] events
was [3] seconds
14/12/18 12:53:27 INFO sink.PhoenixSink: Time taken to process [0] events
was [3] seconds
14/12/18 12:53:33 INFO sink.PhoenixSink: Time taken to process [0] events
was [3] seconds
14/12/18 12:53:40 INFO sink.PhoenixSink: Time taken to process [0] events
was [3] seconds
14/12/18 12:53:48 INFO sink.PhoenixSink: Time taken to process [0] events
was [3] seconds
14/12/18 12:53:56 INFO sink.PhoenixSink: Time taken to process [0] events
was [3] seconds
14/12/18 12:54:04 INFO sink.PhoenixSink: Time taken to process [0] events
was [3] seconds
14/12/18 12:54:12 INFO sink.PhoenixSink: Time taken to process [0] events
was [3] seconds
14/12/18 12:54:20 INFO sink.PhoenixSink: Time taken to process [0] events
was [3] seconds
14/12/18 12:54:28 INFO sink.PhoenixSink: Time taken to process [1] events
was [3] seconds
14/12/18 12:54:36 INFO sink.PhoenixSink: Time taken to process [0] events
was [3] seconds
Thanks
Divya N
