[
https://issues.apache.org/jira/browse/PHOENIX-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431735#comment-15431735
]
Thomas D'Silva edited comment on PHOENIX-3148 at 8/22/16 10:36 PM:
-------------------------------------------------------------------
[~jamestaylor]
I have attached a v3 patch with a "phoenix.serialized.ptable.enabled" option
that is off by default. The median time taken to write 1K rows with and without
serialization while creating the 100K views is 9201ms and x ms. The time taken
to write 1K rows after all the views created are 8902 ms and y ms respectively.
I have also attached a chart showing the comparison.
I am changing the PTable protobuf serialization to serialize/deserialize the
view statement if it is present in the PTable. I think this should not break
backward compatibility right?
{code}
public class PTableImpl implements PTable {
@@ -1073,9 +1072,9 @@ public class PTableImpl implements PTable {
List<PName> physicalNames = Collections.emptyList();
if (tableType == PTableType.VIEW) {
viewType =
ViewType.fromSerializedValue(table.getViewType().toByteArray()[0]);
- if(table.hasViewStatement()){
- viewStatement = (String)
PVarchar.INSTANCE.toObject(table.getViewStatement().toByteArray());
- }
+ }
+ if(table.hasViewStatement()){
+ viewStatement = (String)
PVarchar.INSTANCE.toObject(table.getViewStatement().toByteArray());
}
if (tableType == PTableType.VIEW || viewIndexId != null) {
physicalNames =
Lists.newArrayListWithExpectedSize(table.getPhysicalNamesCount());
@@ -1181,6 +1180,8 @@ public class PTableImpl implements PTable {
builder.setTransactional(table.isTransactional());
if(table.getType() == PTableType.VIEW){
builder.setViewType(ByteStringer.wrap(new
byte[]{table.getViewType().getSerializedValue()}));
+ }
+ if(table.getViewStatement()!=null){
builder.setViewStatement(ByteStringer.wrap(PVarchar.INSTANCE.toBytes(table.getViewStatement())));
}
if(table.getType() == PTableType.VIEW || table.getViewIndexId() != null){
@@ -1244,4 +1245,24 @@ public class PTableImpl implements PTable {
public boolean isAppendOnlySchema() {
return isAppendOnlySchema;
}
{code}
!comparison.png!
was (Author: tdsilva):
[~jamestaylor]
I have attached a v3 patch with a "phoenix.serialized.ptable.enabled" option
that is off by default. The median time taken to write 1K rows with and without
serialization while creating the 100K views is 9201ms and x ms. The time taken
to write 1K rows after all the views created are 8902 ms and y ms respectively.
I have also attached a chart showing the comparison.
I am changing the PTable protobuf serialization to serialize/deserialize the
view statement if it is present in the PTable. I think this should not break
backward compatibility right?
{code}
public class PTableImpl implements PTable {
@@ -1073,9 +1072,9 @@ public class PTableImpl implements PTable {
List<PName> physicalNames = Collections.emptyList();
if (tableType == PTableType.VIEW) {
viewType =
ViewType.fromSerializedValue(table.getViewType().toByteArray()[0]);
- if(table.hasViewStatement()){
- viewStatement = (String)
PVarchar.INSTANCE.toObject(table.getViewStatement().toByteArray());
- }
+ }
+ if(table.hasViewStatement()){
+ viewStatement = (String)
PVarchar.INSTANCE.toObject(table.getViewStatement().toByteArray());
}
if (tableType == PTableType.VIEW || viewIndexId != null) {
physicalNames =
Lists.newArrayListWithExpectedSize(table.getPhysicalNamesCount());
@@ -1181,6 +1180,8 @@ public class PTableImpl implements PTable {
builder.setTransactional(table.isTransactional());
if(table.getType() == PTableType.VIEW){
builder.setViewType(ByteStringer.wrap(new
byte[]{table.getViewType().getSerializedValue()}));
+ }
+ if(table.getViewStatement()!=null){
builder.setViewStatement(ByteStringer.wrap(PVarchar.INSTANCE.toBytes(table.getViewStatement())));
}
if(table.getType() == PTableType.VIEW || table.getViewIndexId() != null){
@@ -1244,4 +1245,24 @@ public class PTableImpl implements PTable {
public boolean isAppendOnlySchema() {
return isAppendOnlySchema;
}
{code}
!comparison.png
> Reduce size of PTable so that more tables can be cached in the metada cache.
> ----------------------------------------------------------------------------
>
> Key: PHOENIX-3148
> URL: https://issues.apache.org/jira/browse/PHOENIX-3148
> Project: Phoenix
> Issue Type: Bug
> Reporter: Thomas D'Silva
> Assignee: Thomas D'Silva
> Labels: argus
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3148-v2.patch, PHOENIX-3148-v3.patch,
> PHOENIX-3148.patch, comparison.png
>
>
> According to PHOENIX-2995, the current size is 7KB per PTable which works out
> to enabling 140K PTables per 1GB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
