H i, I tried with 5 as batchsize,still data is not upserted into phoenix.
14/12/19 12:03:51 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/19 12:03:51 INFO zookeeper.ZooKeeper: Session: 0x14a60b4631e003d closed 14/12/19 12:03:55 INFO serializer.BaseEventSerializer: the upsert statement is UPSERT INTO APACHE_LOG4 ("HOST", "IDENTITY", "USER", "TIME", "METHOD", "REQUEST", "PROTOCOL", "STATUS", "SIZE", "REFERER", "AGENT", "F_HOST", "UID") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] events was [0] seconds 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] events was [0] seconds 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] events was [0] seconds 14/12/19 12:03:58 INFO sink.PhoenixSink: Time taken to process [1] events was [3] seconds 14/12/19 12:04:02 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds 14/12/19 12 Thanks Divya N On Fri, Dec 19, 2014 at 6:10 AM, Ravi Kiran <maghamraviki...@gmail.com> wrote: > > > Hi Nagarajan, > > Apparently, we do batches of 100 by default for each commit . You can > decrease that number if you would like to. > http://phoenix.apache.org/flume.html > > Regards > Ravi > > Hi, >> >> As we are working in flume to store apache log into hbase/phoenix >> using phoenix-flume jars.Using flume conf file we could create table >> in phoenix and see the events get processed.while querying through >> phoenix there is no record stored.please suggest us steps to process >> apache access log as source and store data in phoenix as sink. >> >> Hbase : 0.98.8 >> Phoenix:4.2.1 >> >> Flume Configuration: >> =============== >> >> agent.channels.memory-channel.type = memory >> agent.sources.tail-source.type = exec >> agent.sources.tail-source.command = tail -F /var/log/httpd/access_log >> agent.sources.tail-source.channels = memory-channel >> >> agent.channels.memory-channel.type=memory >> agent.channels.memory-channel.transactionCapacity=100 >> agent.channels.memory-channel.byteCapacityBufferPercentage=20 >> >> >> agent.channels = memory-channel >> agent.sources = tail-source >> agent.sinks = phoenix-sink >> >> >> agent.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink >> agent.sinks.phoenix-sink.channel=memory-channel >> agent.sinks.phoenix-sink.batchSize=100 >> agent.sinks.phoenix-sink.table=APACHE_LOGS >> agent.sinks.phoenix-sink.ddl=CREATE TABLE IF NOT EXISTS >> APACHE_LOGS(uid VARCHAR NOT NULL,host VARCHAR,identity VARCHAR,user >> VARCHAR,time VARCHAR,method VARCHAR,request VARCHAR,protocol >> VARCHAR,status INTEGER,size INTEGER,referer VARCHAR,agent >> VARCHAR,f_host VARCHAR CONSTRAINT pk PRIMARY KEY(uid)) >> agent.sinks.phoenix-sink.zookeeperQuorum=dn02 >> agent.sinks.phoenix-sink.serializer=REGEX >> agent.sinks.phoenix-sink.serializer.regex="^([\\d.]+) (\\S+) (\\S+) >> \\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(.+?)\" (\\d{3}) (\\d+) \"([^\"]+)\" >> \"([^\"]+)\"" >> agent.sinks.phoenix-sink.serializer.rowkeyType=uuid >> >> agent.sinks.phoenix-sink.serializer.columns=host,identity,user,time,method,request,protocol,status,size,referer,agent >> agent.sinks.phoenix-sink.serializer.headers=f_host >> >> flume-ng agent -c conf -f /opt/flume/conf/phoenix.conf -n agent >> >> * >> * >> * >> * >> * >> * >> >> 14/12/18 12:53:09 INFO >> client.HConnectionManager$HConnectionImplementation: Closing master >> protocol: MasterService >> 14/12/18 12:53:09 INFO >> client.HConnectionManager$HConnectionImplementation: Closing zookeeper >> sessionid=0x14a5b888d18003e >> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003e >> closed >> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down >> 14/12/18 12:53:09 INFO query.ConnectionQueryServicesImpl: Found >> quorum: nn01:2181 >> 14/12/18 12:53:09 INFO zookeeper.RecoverableZooKeeper: Process >> identifier=hconnection-0x843735 connecting to ZooKeeper >> ensemble=nn01:2181 >> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Initiating client >> connection, connectString=nn01:2181 sessionTimeout=180000 >> watcher=hconnection-0x843735, quorum=nn01:2181, baseZNode=/hbase >> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Opening socket connection >> to server nn01/10.10.10.25:2181. Will not attempt to authenticate >> using SASL (unknown error) >> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Socket connection >> established to nn01/10.10.10.25:2181, initiating session >> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Session establishment >> complete on server nn01/10.10.10.25:2181, sessionid = >> 0x14a5b888d18003f, negotiated timeout = 40000 >> 14/12/18 12:53:09 INFO >> client.HConnectionManager$HConnectionImplementation: Closing master >> protocol: MasterService >> 14/12/18 12:53:09 INFO >> client.HConnectionManager$HConnectionImplementation: Closing zookeeper >> sessionid=0x14a5b888d18003f >> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003f >> closed >> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down >> 14/12/18 12:53:15 INFO serializer.BaseEventSerializer: the upsert >> statement is UPSERT INTO APACHE_LOG4 ("HOST", "IDENTITY", "USER", >> "TIME", "METHOD", "REQUEST", "PROTOCOL", "STATUS", "SIZE", "REFERER", >> "AGENT", "F_HOST", "UID") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?) >> 14/12/18 12:53:18 INFO sink.PhoenixSink: Time taken to process [10] >> events was [3] seconds >> 14/12/18 12:53:22 INFO sink.PhoenixSink: Time taken to process [0] >> events was [3] seconds >> 14/12/18 12:53:27 INFO sink.PhoenixSink: Time taken to process [0] >> events was [3] seconds >> 14/12/18 12:53:33 INFO sink.PhoenixSink: Time taken to process [0] >> events was [3] seconds >> 14/12/18 12:53:40 INFO sink.PhoenixSink: Time taken to process [0] >> events was [3] seconds >> 14/12/18 12:53:48 INFO sink.PhoenixSink: Time taken to process [0] >> events was [3] seconds >> 14/12/18 12:53:56 INFO sink.PhoenixSink: Time taken to process [0] >> events was [3] seconds >> 14/12/18 12:54:04 INFO sink.PhoenixSink: Time taken to process [0] >> events was [3] seconds >> 14/12/18 12:54:12 INFO sink.PhoenixSink: Time taken to process [0] >> events was [3] seconds >> 14/12/18 12:54:20 INFO sink.PhoenixSink: Time taken to process [0] >> events was [3] seconds >> 14/12/18 12:54:28 INFO sink.PhoenixSink: Time taken to process [1] >> events was [3] seconds >> 14/12/18 12:54:36 INFO sink.PhoenixSink: Time taken to process [0] >> events was [3] seconds >> >> >> Thanks >> >> Divya N >> >