Thanks Sergey. I forgot to mention that the data is loaded via MapReduce. If I load data with HFile, will Phoenix generate the stats automatically? This is what I didn’t understand. I find this on official page "Statistics are also automatically collected during major compactions and region splits so manually running this command may not be necessary."
> On 30 Mar 2018, at 12:44 AM, Sergey Soldatov <sergeysolda...@gmail.com> wrote: > > Usually such kind of problems may happen when something wrong with the > statistic. You may try to clean SYSTEM.STATS, restart the client and check > whether it fixes the problem. If not, you may try to turn on DEBUG log level > on the client and check whether generated scans are covering all regions > (this task is easier to perform with clean stats). > > Thanks, > Sergey > > > On Thu, Mar 29, 2018 at 9:15 AM, Xiaofeng Wang <cap...@gmail.com > <mailto:cap...@gmail.com>> wrote: > Hi, recently I encounter a strange issue about missing data. For instance, > column A, B and C is the primary keys in Phoenix and I can get all the > relevant records via the condition A=‘aaa'. But some time later(maybe one > hour or more), I get the less records than before with the same condition. I > don't know why :( > > BTW: The Phoenix table is salted with bucket 13 > > This is my environment: > > Phoenix: 4.8.0 > HBase: 1.2.0 > > hbase-site.xml: > > <configuration> > <property> > <name>hbase.regionserver.thrift.http</name> > <value>true</value> > </property> > <property> > <name>hbase.thrift.support.proxyuser</name> > <value>true</value> > </property> > <property> > <name>hbase.rootdir</name> > <value>hdfs://ns1/hbase</value> <> > </property> > <property> > <name>hbase.master.port</name> > <value>60000</value> > </property> > <property> > <name>hbase.master.info <http://hbase.master.info/>.port</name> > <value>60010</value> > </property> > <property> > <name>hbase.regionserver.port</name> > <value>60020</value> > </property> > <property> > <name>hbase.regionserver.info.port</name> > <value>60030</value> > </property> > <property> > <name>hbase.cluster.distributed</name> > <value>true</value> > </property> > <property> > <name>hbase.zookeeper.quorum</name> > <value>host1:2181,host2:2181,host3:2181</value> > </property> > <property> > <name>zookeeper.recovery.retry</name> > <value>3</value> > </property> > <property> > <name>zookeeper.session.timeout</name> > <value>60000</value> > </property> > <property> > <name>hbase.ipc.server.callqueue.handler.factor</name> > <value>0.2</value> > </property> > <property> > <name>hbase.ipc.server.callqueue.read.ratio</name> > <value>0.3</value> > </property> > <property> > <name>hbase.ipc.server.callqueue.scan.ratio</name> > <value>0.4</value> > </property> > <property> > <name>hbase.client.max.perserver.tasks</name> > <value>7</value> > </property> > <property> > <name>hbase.client.max.perregion.tasks</name> > <value>4</value> > </property> > <property> > <name>hbase.offpeak.start.hour</name> > <value>2</value> > </property> > <property> > <name>hbase.offpeak.end.hour</name> > <value>4</value> > </property> > <property> > <name>hbase.storescanner.parallel.seek.enable</name> > <value>true</value> > </property> > <property> > <name>hbase.storescanner.parallel.seek.threads</name> > <value>10</value> > </property> > <property> > <name>hbase.client.retries.number</name> > <value>5</value> > </property> > <property> > <name>hbase.client.scanner.caching</name> > <value>1000</value> > </property> > <property> > <name>hbase.regionserver.handler.count</name> > <value>100</value> > </property> > <property> > <name>hfile.block.cache.size</name> > <value>0.4</value> > </property> > <property> > <name>hbase.coprocessor.abortonerror</name> > <value>false</value> > </property> > <property> > <name>hbase.regionserver.thrift.framed</name> > <value>false</value> > </property> > <property> > <name>hbase.column.max.version</name> > <value>3</value> > </property> > <property> > <name>hbase.status.published</name> > <value>false</value> > </property> > <property> > <name>hbase.status.multicast.address.port</name> > <value>61000</value> > </property> > <property> > <name>hbase.lease.recovery.dfs.timeout</name> > <value>2300000</value> > </property> > <property> > <name>dfs.client.sockettimeout</name> > <value>1000000</value> > </property> > <property> > <name>hbase.master.distributed.log.replay</name> > <value>true</value> > </property> > <property> > <name>hbase.rpc.engine</name> > <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> > </property> > <property> > <name>hbase.coprocessor.master.classes</name> > <value>org.apache.hadoop.hbase.security.access.AccessController</value> > </property> > <property> > <name>hbase.coprocessor.region.classes</name> > <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value> > </property> > <property> > <name>hbase.superuser</name> > <value>hbase</value> > </property> > <property> > <name>hbase.security.authorization</name> > <value>false</value> > </property> > <property> > <name>hbase.client.write.buffer</name> > <value>5242880</value> > </property> > <property> > <name>hbase.hregion.max.filesize</name> > <value>10737418240</value> > </property> > <property> > <name>hbase.rpc.timeout</name> > <value>1000000</value> > </property> > <property> > <name>hbase.regionserver.wal.codec</name> > <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value> > </property> > <property> > <name>hbase.client.keyvalue.maxsize</name> > <value>20971520</value> > </property> > <property> > <name>hbase.hregion.memstore.flush.size</name> > <value>536870912</value> > </property> > <property> > <name>hbase.hregion.memstore.block.multiplier</name> > <value>8</value> > </property> > <property> > <name>hbase.hstore.blockingStoreFiles</name> > <value>300</value> > </property> > <property> > <name>hbase.hstore.compactionThreshold</name> > <value>10</value> > </property> > <property> > <name>hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily</name> > <value>320</value> > </property> > </configuration> > > Thank you all! >
