[
https://issues.apache.org/jira/browse/PHOENIX-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431735#comment-15431735
]
Thomas D'Silva edited comment on PHOENIX-3148 at 8/22/16 10:37 PM:
-------------------------------------------------------------------
[~jamestaylor]
I have attached a v3 patch with a "phoenix.serialized.ptable.enabled" option
that is off by default. The median time taken to write 1K rows with and without
serialization while creating the 100K views is 9201ms and 8715.5 ms. The time
taken to write 1K rows after all the views created are 8902 ms and 267 ms
respectively. I have also attached a chart showing the comparison.
I am changing the PTable protobuf serialization to serialize/deserialize the
view statement if it is present in the PTable. I think this should not break
backward compatibility right?
{code}
public class PTableImpl implements PTable {
@@ -1073,9 +1072,9 @@ public class PTableImpl implements PTable {
List<PName> physicalNames = Collections.emptyList();
if (tableType == PTableType.VIEW) {
viewType =
ViewType.fromSerializedValue(table.getViewType().toByteArray()[0]);
- if(table.hasViewStatement()){
- viewStatement = (String)
PVarchar.INSTANCE.toObject(table.getViewStatement().toByteArray());
- }
+ }
+ if(table.hasViewStatement()){
+ viewStatement = (String)
PVarchar.INSTANCE.toObject(table.getViewStatement().toByteArray());
}
if (tableType == PTableType.VIEW || viewIndexId != null) {
physicalNames =
Lists.newArrayListWithExpectedSize(table.getPhysicalNamesCount());
@@ -1181,6 +1180,8 @@ public class PTableImpl implements PTable {
builder.setTransactional(table.isTransactional());
if(table.getType() == PTableType.VIEW){
builder.setViewType(ByteStringer.wrap(new
byte[]{table.getViewType().getSerializedValue()}));
+ }
+ if(table.getViewStatement()!=null){
builder.setViewStatement(ByteStringer.wrap(PVarchar.INSTANCE.toBytes(table.getViewStatement())));
}
if(table.getType() == PTableType.VIEW || table.getViewIndexId() != null){
@@ -1244,4 +1245,24 @@ public class PTableImpl implements PTable {
public boolean isAppendOnlySchema() {
return isAppendOnlySchema;
}
{code}
!comparison.png!
was (Author: tdsilva):
[~jamestaylor]
I have attached a v3 patch with a "phoenix.serialized.ptable.enabled" option
that is off by default. The median time taken to write 1K rows with and without
serialization while creating the 100K views is 9201ms and x ms. The time taken
to write 1K rows after all the views created are 8902 ms and y ms respectively.
I have also attached a chart showing the comparison.
I am changing the PTable protobuf serialization to serialize/deserialize the
view statement if it is present in the PTable. I think this should not break
backward compatibility right?
{code}
public class PTableImpl implements PTable {
@@ -1073,9 +1072,9 @@ public class PTableImpl implements PTable {
List<PName> physicalNames = Collections.emptyList();
if (tableType == PTableType.VIEW) {
viewType =
ViewType.fromSerializedValue(table.getViewType().toByteArray()[0]);
- if(table.hasViewStatement()){
- viewStatement = (String)
PVarchar.INSTANCE.toObject(table.getViewStatement().toByteArray());
- }
+ }
+ if(table.hasViewStatement()){
+ viewStatement = (String)
PVarchar.INSTANCE.toObject(table.getViewStatement().toByteArray());
}
if (tableType == PTableType.VIEW || viewIndexId != null) {
physicalNames =
Lists.newArrayListWithExpectedSize(table.getPhysicalNamesCount());
@@ -1181,6 +1180,8 @@ public class PTableImpl implements PTable {
builder.setTransactional(table.isTransactional());
if(table.getType() == PTableType.VIEW){
builder.setViewType(ByteStringer.wrap(new
byte[]{table.getViewType().getSerializedValue()}));
+ }
+ if(table.getViewStatement()!=null){
builder.setViewStatement(ByteStringer.wrap(PVarchar.INSTANCE.toBytes(table.getViewStatement())));
}
if(table.getType() == PTableType.VIEW || table.getViewIndexId() != null){
@@ -1244,4 +1245,24 @@ public class PTableImpl implements PTable {
public boolean isAppendOnlySchema() {
return isAppendOnlySchema;
}
{code}
!comparison.png!
> Reduce size of PTable so that more tables can be cached in the metada cache.
> ----------------------------------------------------------------------------
>
> Key: PHOENIX-3148
> URL: https://issues.apache.org/jira/browse/PHOENIX-3148
> Project: Phoenix
> Issue Type: Bug
> Reporter: Thomas D'Silva
> Assignee: Thomas D'Silva
> Labels: argus
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3148-v2.patch, PHOENIX-3148-v3.patch,
> PHOENIX-3148.patch, comparison.png
>
>
> According to PHOENIX-2995, the current size is 7KB per PTable which works out
> to enabling 140K PTables per 1GB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
