[jira] [Commented] (KUDU-3201) Support gzipped HMS notifications (GzipJSONMessageEncoder)
[ https://issues.apache.org/jira/browse/KUDU-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17211417#comment-17211417 ] ASF subversion and git services commented on KUDU-3201: --- Commit 6b20440f4c51a6b69c1382db51139bf8d3467b05 in kudu's branch refs/heads/master from Grant Henke [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=6b20440 ] KUDU-3201: [hms] Support gzipped HMS notifications (GzipJSONMessageEncoder) This patch adds support for HMS notification messages encoded via the GzipJSONMessageEncoder that was added in HIVE-20679. It does this by decoding the base64 encoding and decompressing the messages sent with the `gzip(json-2.0)` format. Because HIVE-20679 is only in Hive 4 which is unreleased, I created a static encoded message to test the gzip functionality. I also manually tested this functionality on a real cluster that has HIVE-20679. While testing this patch I found and fixed a bug in our zlib.cc Uncompress method. Without the fix zlib would return a BUF_ERROR when the decompressed data fits in the output buffer on the first call to `inflate`. For reference I used the following post to help understand the expected usage of `inflate`: https://zlib.net/zlib_how.html Change-Id: I0f94e2446255e1fafb1dac9bf9ce23b81d6c0960 Reviewed-on: http://gerrit.cloudera.org:8080/16565 Tested-by: Grant Henke Reviewed-by: Andrew Wong > Support gzipped HMS notifications (GzipJSONMessageEncoder) > -- > > Key: KUDU-3201 > URL: https://issues.apache.org/jira/browse/KUDU-3201 > Project: Kudu > Issue Type: Improvement > Components: hms >Affects Versions: 1.13.0 >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > > In HIVE-20679 a new message encoder was added to Hive that gzips and based64 > encodes notification messages. In HIVE-21307 the default encoder was changed > to use the gzip encoder, GzipJSONMessageEncoder. > Though Hive 4 is not released, failures as a result of this new message > format were reported in IMPALA-8751. We should add support for the gzip > format to prevent future HMS sync issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3201) Support gzipped HMS notifications (GzipJSONMessageEncoder)
[ https://issues.apache.org/jira/browse/KUDU-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3201. --- Fix Version/s: 1.14.0 Resolution: Fixed > Support gzipped HMS notifications (GzipJSONMessageEncoder) > -- > > Key: KUDU-3201 > URL: https://issues.apache.org/jira/browse/KUDU-3201 > Project: Kudu > Issue Type: Improvement > Components: hms >Affects Versions: 1.13.0 >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Fix For: 1.14.0 > > > In HIVE-20679 a new message encoder was added to Hive that gzips and based64 > encodes notification messages. In HIVE-21307 the default encoder was changed > to use the gzip encoder, GzipJSONMessageEncoder. > Though Hive 4 is not released, failures as a result of this new message > format were reported in IMPALA-8751. We should add support for the gzip > format to prevent future HMS sync issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KUDU-3197) Tablet keeps all history schemas in memory may result in high memory consumption
[
https://issues.apache.org/jira/browse/KUDU-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17211340#comment-17211340
]
Andrew Wong edited comment on KUDU-3197 at 10/9/20, 8:16 PM:
-
You're right in that the scanner creates a schema to represent its projection.
However, the underlying iterators may take references to the current schema
while iterating, the tablet service might take references while preparing to
scan, etc. While most if not all of these accesses are short-lived, we need to
be careful not to destruct the schemas while these references still exist.
Grepping around to audit current usages (with some false positives for copies
and log messages):
{code:java}
~/Repositories/kudu/src/kudu > grep -r "meta.*[^_]schema()" .
./tablet/tablet-schema-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet-schema-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet-schema-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet-schema-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet-schema-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet_metadata.cc:if (!(*metadata)->schema().Equals(schema)) {
./tablet/tablet_metadata.cc:"match expected schema ($1)",
(*metadata)->schema().ToString(),
./tablet/diff_scan-test.cc: SchemaBuilder
builder(tablet->metadata()->schema());
./tablet/tablet_replica-test.cc:
ASSERT_OK(SchemaToPB(SchemaBuilder(tablet()->metadata()->schema()).Build(),
&orig_schema_pb));
./tablet/tablet_replica-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet_replica-test.cc:
ASSERT_OK(SchemaToPB(SchemaBuilder(tablet()->metadata()->schema()).Build(),
&orig_schema_pb));
./tablet/tablet_replica-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet.cc: : key_schema_(metadata->schema().CreateKeyProjection()),
./tablet/tablet.cc: metadata_->SetSchema(*op_state->schema(),
op_state->schema_version());
./tablet/tablet.cc: RollingDiskRowSetWriter drsw(metadata_.get(),
merge->schema(), DefaultBloomSizing(),
./tablet/rowset_metadata.h:return tablet_metadata_->schema();
./tablet/tablet.h:return &metadata_->schema();
./tablet/all_types-scan-correctness-test.cc:SchemaBuilder
builder(tablet()->metadata()->schema());
./tools/kudu-tool-test.cc: .PartitionDebugString(meta->partition(),
meta->schema());
./tools/kudu-tool-test.cc: debug_str = meta->schema().ToString();
./tools/kudu-tool-test.cc:.PartitionDebugString(meta->partition(),
meta->schema());
./tools/kudu-tool-test.cc:debug_str = meta->schema().ToString();
./tools/tool_action_local_replica.cc:const auto& col_idx =
meta->schema().find_column_by_id(col_id);
./tools/tool_action_local_replica.cc:
meta->schema().column(col_idx).name() : "?");
./tools/tool_action_local_replica.cc: Schema schema = meta->schema();
./tools/tool_action_local_replica.cc: const Schema& schema = meta->schema();
./tools/tool_action_local_replica.cc:
meta->schema())
./tserver/tablet_service.cc:Schema tablet_schema =
replica->tablet_metadata()->schema();
./tserver/tablet_service.cc: const auto& schema =
replica->tablet_metadata()->schema();
./tserver/tablet_service.cc:
CHECK_OK(SchemaToPB(replica->tablet_metadata()->schema(),
./tserver/tablet_service.cc: const auto& schema =
replica->tablet_metadata()->schema();
./tserver/tablet_service.cc: Schema tablet_schema =
replica->tablet_metadata()->schema();
./tserver/tablet_service.cc: const auto& schema =
replica->tablet_metadata()->schema();
./tserver/tablet_service.cc: const Schema& tablet_schema =
replica->tablet_metadata()->schema();
./tserver/scanners.cc:
spec().lower_bound_key()->Stringify(tablet_metadata->schema();
./tserver/scanners.cc:
spec().exclusive_upper_bound_key()->Stringify(tablet_metadata->schema();
./tserver/tserver_path_handlers.cc:
tmeta->schema());
./tserver/tserver_path_handlers.cc: const Schema& schema = tmeta->schema();
./master/sys_catalog.cc: if (!metadata->schema().Equals(BuildTableSchema())) {
./master/sys_catalog.cc:return(Status::Corruption("Unexpected schema",
metadata->schema().ToString()));
./client/scan_token-internal.cc:
RETURN_NOT_OK(SchemaFromPB(metadata.schema(), &schema));
./client/client-test.cc:Schema schema =
tablet_replica->tablet()->metadata()->schema();
./client/client-test.cc:Schema schema =
tablet_replica->tablet()->metadata()->schema();
./client/client-test.cc:Schema schema =
tablet_replica->tablet()->metadata()->schema();
./client/client-test.cc:Schema schema =
tablet_replica->tablet()->metada
[jira] [Commented] (KUDU-3197) Tablet keeps all history schemas in memory may result in high memory consumption
[
https://issues.apache.org/jira/browse/KUDU-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17211340#comment-17211340
]
Andrew Wong commented on KUDU-3197:
---
You're right in that the scanner creates a schema to represent its projection.
However, the underlying iterators may take references to the current schema
while iterating, the tablet service might take references while preparing to
scan, etc. While most if not all of these accesses are short-lived, we need to
be careful not to destruct the schemas while these references still exist.
Grepping around to audit current usages (with some false positives for copies
and log messages):
{code:java}
~/Repositories/kudu/src/kudu > grep -r "meta.*[^_]schema()" .
./tablet/tablet-schema-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet-schema-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet-schema-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet-schema-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet-schema-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet_metadata.cc:if (!(*metadata)->schema().Equals(schema)) {
./tablet/tablet_metadata.cc:"match expected schema ($1)",
(*metadata)->schema().ToString(),
./tablet/diff_scan-test.cc: SchemaBuilder
builder(tablet->metadata()->schema());
./tablet/tablet_replica-test.cc:
ASSERT_OK(SchemaToPB(SchemaBuilder(tablet()->metadata()->schema()).Build(),
&orig_schema_pb));
./tablet/tablet_replica-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet_replica-test.cc:
ASSERT_OK(SchemaToPB(SchemaBuilder(tablet()->metadata()->schema()).Build(),
&orig_schema_pb));
./tablet/tablet_replica-test.cc: SchemaBuilder
builder(tablet()->metadata()->schema());
./tablet/tablet.cc: : key_schema_(metadata->schema().CreateKeyProjection()),
./tablet/tablet.cc: metadata_->SetSchema(*op_state->schema(),
op_state->schema_version());
./tablet/tablet.cc: RollingDiskRowSetWriter drsw(metadata_.get(),
merge->schema(), DefaultBloomSizing(),
./tablet/rowset_metadata.h:return tablet_metadata_->schema();
./tablet/tablet.h:return &metadata_->schema();
./tablet/all_types-scan-correctness-test.cc:SchemaBuilder
builder(tablet()->metadata()->schema());
./tools/kudu-tool-test.cc: .PartitionDebugString(meta->partition(),
meta->schema());
./tools/kudu-tool-test.cc: debug_str = meta->schema().ToString();
./tools/kudu-tool-test.cc:.PartitionDebugString(meta->partition(),
meta->schema());
./tools/kudu-tool-test.cc:debug_str = meta->schema().ToString();
./tools/tool_action_local_replica.cc:const auto& col_idx =
meta->schema().find_column_by_id(col_id);
./tools/tool_action_local_replica.cc:
meta->schema().column(col_idx).name() : "?");
./tools/tool_action_local_replica.cc: Schema schema = meta->schema();
./tools/tool_action_local_replica.cc: const Schema& schema = meta->schema();
./tools/tool_action_local_replica.cc:
meta->schema())
./tserver/tablet_service.cc:Schema tablet_schema =
replica->tablet_metadata()->schema();
./tserver/tablet_service.cc: const auto& schema =
replica->tablet_metadata()->schema();
./tserver/tablet_service.cc:
CHECK_OK(SchemaToPB(replica->tablet_metadata()->schema(),
./tserver/tablet_service.cc: const auto& schema =
replica->tablet_metadata()->schema();
./tserver/tablet_service.cc: Schema tablet_schema =
replica->tablet_metadata()->schema();
./tserver/tablet_service.cc: const auto& schema =
replica->tablet_metadata()->schema();
./tserver/tablet_service.cc: const Schema& tablet_schema =
replica->tablet_metadata()->schema();
./tserver/scanners.cc:
spec().lower_bound_key()->Stringify(tablet_metadata->schema();
./tserver/scanners.cc:
spec().exclusive_upper_bound_key()->Stringify(tablet_metadata->schema();
./tserver/tserver_path_handlers.cc:
tmeta->schema());
./tserver/tserver_path_handlers.cc: const Schema& schema = tmeta->schema();
./master/sys_catalog.cc: if (!metadata->schema().Equals(BuildTableSchema())) {
./master/sys_catalog.cc:return(Status::Corruption("Unexpected schema",
metadata->schema().ToString()));
./client/scan_token-internal.cc:
RETURN_NOT_OK(SchemaFromPB(metadata.schema(), &schema));
./client/client-test.cc:Schema schema =
tablet_replica->tablet()->metadata()->schema();
./client/client-test.cc:Schema schema =
tablet_replica->tablet()->metadata()->schema();
./client/client-test.cc:Schema schema =
tablet_replica->tablet()->metadata()->schema();
./client/client-test.cc:Schema schema =
tablet_replica->tablet()->metadata()->schema();
./client/client-test.cc:Schema
[jira] [Resolved] (KUDU-3198) Unable to delete a full row from a table with 64 columns when using java client
[
https://issues.apache.org/jira/browse/KUDU-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Serbin resolved KUDU-3198.
-
Resolution: Fixed
Fixed with the patch mentioned in the prior comment.
Thank you [~zhangyifan27] for debugging and fixing the issue!
> Unable to delete a full row from a table with 64 columns when using java
> client
> ---
>
> Key: KUDU-3198
> URL: https://issues.apache.org/jira/browse/KUDU-3198
> Project: Kudu
> Issue Type: Bug
> Components: java
>Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.12.0, 1.11.1, 1.13.0
>Reporter: YifanZhang
>Priority: Major
> Fix For: 1.14.0
>
>
> We recently got an error when deleted full rows from a table with 64 columns
> using sparkSQL, however if we delete a column from the table, this error will
> not appear. The error is:
> {code:java}
> Failed to write at least 1000 rows to Kudu; Sample errors: Not implemented:
> Unknown row operation type (error 0){code}
> I tested this by deleting a full row from a table with 64 column using java
> client 1.12.0/1.13.0, if the row is set NULL for some columns, I got an error:
> {code:java}
> Row error for primary key=[-128, 0, 0, 1], tablet=null,
> server=d584b3407ea444519e91b32f2744b162, status=Invalid argument: DELETE
> should not have a value for column: c63 STRING NULLABLE (error 0)
> {code}
> if the row is set values for all columns , I got an error like:
> {code:java}
> Row error for primary key=[-128, 0, 0, 1], tablet=null, server=null,
> status=Corruption: Not enough data for column: c63 STRING NULLABLE (error 0)
> {code}
> I also tested this with tables with different number of columns. The weird
> thing is I could delete full rows from a table with 8/16/32/63/65 columns,
> but couldn't do this if the table has 64/128 columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3198) Unable to delete a full row from a table with 64 columns when using java client
[
https://issues.apache.org/jira/browse/KUDU-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Serbin updated KUDU-3198:
Fix Version/s: 1.14.0
> Unable to delete a full row from a table with 64 columns when using java
> client
> ---
>
> Key: KUDU-3198
> URL: https://issues.apache.org/jira/browse/KUDU-3198
> Project: Kudu
> Issue Type: Bug
> Components: java
>Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.12.0, 1.11.1, 1.13.0
>Reporter: YifanZhang
>Priority: Major
> Fix For: 1.14.0
>
>
> We recently got an error when deleted full rows from a table with 64 columns
> using sparkSQL, however if we delete a column from the table, this error will
> not appear. The error is:
> {code:java}
> Failed to write at least 1000 rows to Kudu; Sample errors: Not implemented:
> Unknown row operation type (error 0){code}
> I tested this by deleting a full row from a table with 64 column using java
> client 1.12.0/1.13.0, if the row is set NULL for some columns, I got an error:
> {code:java}
> Row error for primary key=[-128, 0, 0, 1], tablet=null,
> server=d584b3407ea444519e91b32f2744b162, status=Invalid argument: DELETE
> should not have a value for column: c63 STRING NULLABLE (error 0)
> {code}
> if the row is set values for all columns , I got an error like:
> {code:java}
> Row error for primary key=[-128, 0, 0, 1], tablet=null, server=null,
> status=Corruption: Not enough data for column: c63 STRING NULLABLE (error 0)
> {code}
> I also tested this with tables with different number of columns. The weird
> thing is I could delete full rows from a table with 8/16/32/63/65 columns,
> but couldn't do this if the table has 64/128 columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3198) Unable to delete a full row from a table with 64 columns when using java client
[
https://issues.apache.org/jira/browse/KUDU-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Serbin updated KUDU-3198:
Affects Version/s: 1.10.0
1.10.1
1.11.0
1.11.1
> Unable to delete a full row from a table with 64 columns when using java
> client
> ---
>
> Key: KUDU-3198
> URL: https://issues.apache.org/jira/browse/KUDU-3198
> Project: Kudu
> Issue Type: Bug
> Components: java
>Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.12.0, 1.11.1, 1.13.0
>Reporter: YifanZhang
>Priority: Major
>
> We recently got an error when deleted full rows from a table with 64 columns
> using sparkSQL, however if we delete a column from the table, this error will
> not appear. The error is:
> {code:java}
> Failed to write at least 1000 rows to Kudu; Sample errors: Not implemented:
> Unknown row operation type (error 0){code}
> I tested this by deleting a full row from a table with 64 column using java
> client 1.12.0/1.13.0, if the row is set NULL for some columns, I got an error:
> {code:java}
> Row error for primary key=[-128, 0, 0, 1], tablet=null,
> server=d584b3407ea444519e91b32f2744b162, status=Invalid argument: DELETE
> should not have a value for column: c63 STRING NULLABLE (error 0)
> {code}
> if the row is set values for all columns , I got an error like:
> {code:java}
> Row error for primary key=[-128, 0, 0, 1], tablet=null, server=null,
> status=Corruption: Not enough data for column: c63 STRING NULLABLE (error 0)
> {code}
> I also tested this with tables with different number of columns. The weird
> thing is I could delete full rows from a table with 8/16/32/63/65 columns,
> but couldn't do this if the table has 64/128 columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-3198) Unable to delete a full row from a table with 64 columns when using java client
[
https://issues.apache.org/jira/browse/KUDU-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17211307#comment-17211307
]
ASF subversion and git services commented on KUDU-3198:
---
Commit ea48c9b181e3256cb96940c8d6bbf2f5ec3c3607 in kudu's branch
refs/heads/master from zhangyifan27
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=ea48c9b ]
KUDU-3198: fix encodeRow() when encoding delete operations
Before this patch, not all non-key columns are cleared when
encoding delete row operations. If users delele with full row
from a 64 column table, the operation would fail.
Change-Id: I144e1998b99be3e660b7f320c55e4b1c9753a61f
Reviewed-on: http://gerrit.cloudera.org:8080/16571
Reviewed-by: Grant Henke
Tested-by: Grant Henke
Reviewed-by: Alexey Serbin
> Unable to delete a full row from a table with 64 columns when using java
> client
> ---
>
> Key: KUDU-3198
> URL: https://issues.apache.org/jira/browse/KUDU-3198
> Project: Kudu
> Issue Type: Bug
> Components: java
>Affects Versions: 1.12.0, 1.13.0
>Reporter: YifanZhang
>Priority: Major
>
> We recently got an error when deleted full rows from a table with 64 columns
> using sparkSQL, however if we delete a column from the table, this error will
> not appear. The error is:
> {code:java}
> Failed to write at least 1000 rows to Kudu; Sample errors: Not implemented:
> Unknown row operation type (error 0){code}
> I tested this by deleting a full row from a table with 64 column using java
> client 1.12.0/1.13.0, if the row is set NULL for some columns, I got an error:
> {code:java}
> Row error for primary key=[-128, 0, 0, 1], tablet=null,
> server=d584b3407ea444519e91b32f2744b162, status=Invalid argument: DELETE
> should not have a value for column: c63 STRING NULLABLE (error 0)
> {code}
> if the row is set values for all columns , I got an error like:
> {code:java}
> Row error for primary key=[-128, 0, 0, 1], tablet=null, server=null,
> status=Corruption: Not enough data for column: c63 STRING NULLABLE (error 0)
> {code}
> I also tested this with tables with different number of columns. The weird
> thing is I could delete full rows from a table with 8/16/32/63/65 columns,
> but couldn't do this if the table has 64/128 columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3198) Unable to delete a full row from a table with 64 columns when using java client
[
https://issues.apache.org/jira/browse/KUDU-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YifanZhang updated KUDU-3198:
-
Description:
We recently got an error when deleted full rows from a table with 64 columns
using sparkSQL, however if we delete a column from the table, this error will
not appear. The error is:
{code:java}
Failed to write at least 1000 rows to Kudu; Sample errors: Not implemented:
Unknown row operation type (error 0){code}
I tested this by deleting a full row from a table with 64 column using java
client 1.12.0/1.13.0, if the row is set NULL for some columns, I got an error:
{code:java}
Row error for primary key=[-128, 0, 0, 1], tablet=null,
server=d584b3407ea444519e91b32f2744b162, status=Invalid argument: DELETE should
not have a value for column: c63 STRING NULLABLE (error 0)
{code}
if the row is set values for all columns , I got an error like:
{code:java}
Row error for primary key=[-128, 0, 0, 1], tablet=null, server=null,
status=Corruption: Not enough data for column: c63 STRING NULLABLE (error 0)
{code}
I also tested this with tables with different number of columns. The weird
thing is I could delete full rows from a table with 8/16/32/63/65 columns, but
couldn't do this if the table has 64/128 columns.
was:
We recently got an error when deleted full rows from a table with 64 columns
using sparkSQL, however if we delete a column, this error will not appear. The
error is:
{code:java}
Failed to write at least 1000 rows to Kudu; Sample errors: Not implemented:
Unknown row operation type (error 0){code}
I tested this by deleting a full row from a table with 64 column using java
client 1.12.0/1.13.0, if the row is set NULL for some columns, I got an error:
{code:java}
Row error for primary key=[-128, 0, 0, 1], tablet=null,
server=d584b3407ea444519e91b32f2744b162, status=Invalid argument: DELETE should
not have a value for column: c63 STRING NULLABLE (error 0)
{code}
if the row is set values for all columns , I got an error like:
{code:java}
Row error for primary key=[-128, 0, 0, 1], tablet=null, server=null,
status=Corruption: Not enough data for column: c63 STRING NULLABLE (error 0)
{code}
I also tested this with tables with different number of columns. The weird
thing is I could delete full rows from a table with 8/16/32/63/65 columns, but
couldn't do this if the table has 64/128 columns.
> Unable to delete a full row from a table with 64 columns when using java
> client
> ---
>
> Key: KUDU-3198
> URL: https://issues.apache.org/jira/browse/KUDU-3198
> Project: Kudu
> Issue Type: Bug
> Components: java
>Affects Versions: 1.12.0, 1.13.0
>Reporter: YifanZhang
>Priority: Major
>
> We recently got an error when deleted full rows from a table with 64 columns
> using sparkSQL, however if we delete a column from the table, this error will
> not appear. The error is:
> {code:java}
> Failed to write at least 1000 rows to Kudu; Sample errors: Not implemented:
> Unknown row operation type (error 0){code}
> I tested this by deleting a full row from a table with 64 column using java
> client 1.12.0/1.13.0, if the row is set NULL for some columns, I got an error:
> {code:java}
> Row error for primary key=[-128, 0, 0, 1], tablet=null,
> server=d584b3407ea444519e91b32f2744b162, status=Invalid argument: DELETE
> should not have a value for column: c63 STRING NULLABLE (error 0)
> {code}
> if the row is set values for all columns , I got an error like:
> {code:java}
> Row error for primary key=[-128, 0, 0, 1], tablet=null, server=null,
> status=Corruption: Not enough data for column: c63 STRING NULLABLE (error 0)
> {code}
> I also tested this with tables with different number of columns. The weird
> thing is I could delete full rows from a table with 8/16/32/63/65 columns,
> but couldn't do this if the table has 64/128 columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (KUDU-3197) Tablet keeps all history schemas in memory may result in high memory consumption
[ https://issues.apache.org/jira/browse/KUDU-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangningito updated KUDU-3197: -- Comment: was deleted (was: With deeper look into existing code, I get a little doubt with the scanners. I saw two methods in tablet_service.cc, `HandleNewScanRequest` and `HandleContinueScanRequest` charges almost all scan requests. And it use projection schema to serialize the scan result. In `HandleNewScanRequest` it create projection schema of scanner in scanner manager, and serialize the result by the projection. In `HandleContinueScanRequest`, it get scanner with scanner_id from scanner manager, the projection schema is kept in scanner, so it may be irrelevant to the schema kept by tablet_metadata. So I'm wondering it may not have to much impact on deleting old old_schemas_. BTW, it took me some time to thinking about ref-counted approach, as I understand, I considered two approach, 1. Change change schema_ kept by tablet_metadata to shared_ptr wrapped one, and add a lock in SetSchema() and Schema() cause previously atomic swap is broken. It may hurts the perform a lot. 2. Add a atomic counter as Schema field, and decrease ref in scanner dtor, it may also hurt the performance when getting Schema(). ) > Tablet keeps all history schemas in memory may result in high memory > consumption > > > Key: KUDU-3197 > URL: https://issues.apache.org/jira/browse/KUDU-3197 > Project: Kudu > Issue Type: Improvement > Components: tablet >Affects Versions: 1.12.0 >Reporter: wangningito >Assignee: wangningito >Priority: Minor > Attachments: image-2020-09-25-14-45-33-402.png, > image-2020-09-25-14-49-30-913.png, image-2020-09-25-15-05-44-948.png > > > In case of high frequency of updating table, memory consumption of > kudu-tserver may be very high, and the memory in not tracked in the memory > page. > This is the memory usage of a tablet, the memory consumption of tablet-xxx‘s > peak is 3.6G， but none of its' childrens' memory can reach. > !image-2020-09-25-14-45-33-402.png! > So I use pprof to get the heap sampling. The tserver started for long but the > memory is still consuming by TabletBootstrap:PlayAlterSchemaRequest. > !image-2020-09-25-14-49-30-913.png! > I change the `old_schemas_` in tablet_metadata.h to a fixed size vector, > // Previous values of 'schema_'. > // These are currently kept alive forever, under the assumption that > // a given tablet won't have thousands of "alter table" calls. > // They are kept alive so that callers of schema() don't need to > // worry about reference counting or locking. > std::vector old_schemas_; > The heap sampling then becomes > !image-2020-09-25-15-05-44-948.png! > So, to make application layer more flexible, it could be better to make the > size of the old_schemas configurable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-3197) Tablet keeps all history schemas in memory may result in high memory consumption
[ https://issues.apache.org/jira/browse/KUDU-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210744#comment-17210744 ] wangningito commented on KUDU-3197: --- With deeper look into existing code, I get a little doubt with the scanners. I saw two methods in tablet_service.cc, `HandleNewScanRequest` and `HandleContinueScanRequest` charges almost all scan requests. And it use projection schema to serialize the scan result. In `HandleNewScanRequest` it create projection schema of scanner in scanner manager, and serialize the result by the projection. In `HandleContinueScanRequest`, it get scanner with scanner_id from scanner manager, the projection schema is kept in scanner, so it may be irrelevant to the schema kept by tablet_metadata. So I'm wondering it may not have to much impact on deleting old old_schemas_. BTW, it took me some time to thinking about ref-counted approach, as I understand, I considered two approach, 1. Change change schema_ kept by tablet_metadata to shared_ptr wrapped one, and add a lock in SetSchema() and Schema() cause previously atomic swap is broken. It may hurts the perform a lot. 2. Add a atomic counter as Schema field, and decrease ref in scanner dtor, it may also hurt the performance when getting Schema(). > Tablet keeps all history schemas in memory may result in high memory > consumption > > > Key: KUDU-3197 > URL: https://issues.apache.org/jira/browse/KUDU-3197 > Project: Kudu > Issue Type: Improvement > Components: tablet >Affects Versions: 1.12.0 >Reporter: wangningito >Assignee: wangningito >Priority: Minor > Attachments: image-2020-09-25-14-45-33-402.png, > image-2020-09-25-14-49-30-913.png, image-2020-09-25-15-05-44-948.png > > > In case of high frequency of updating table, memory consumption of > kudu-tserver may be very high, and the memory in not tracked in the memory > page. > This is the memory usage of a tablet, the memory consumption of tablet-xxx‘s > peak is 3.6G， but none of its' childrens' memory can reach. > !image-2020-09-25-14-45-33-402.png! > So I use pprof to get the heap sampling. The tserver started for long but the > memory is still consuming by TabletBootstrap:PlayAlterSchemaRequest. > !image-2020-09-25-14-49-30-913.png! > I change the `old_schemas_` in tablet_metadata.h to a fixed size vector, > // Previous values of 'schema_'. > // These are currently kept alive forever, under the assumption that > // a given tablet won't have thousands of "alter table" calls. > // They are kept alive so that callers of schema() don't need to > // worry about reference counting or locking. > std::vector old_schemas_; > The heap sampling then becomes > !image-2020-09-25-15-05-44-948.png! > So, to make application layer more flexible, it could be better to make the > size of the old_schemas configurable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-3197) Tablet keeps all history schemas in memory may result in high memory consumption
[ https://issues.apache.org/jira/browse/KUDU-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210745#comment-17210745 ] wangningito commented on KUDU-3197: --- With deeper look into existing code, I get a little doubt with the scanners. I saw two methods in tablet_service.cc, `HandleNewScanRequest` and `HandleContinueScanRequest` charges almost all scan requests. And it use projection schema to serialize the scan result. In `HandleNewScanRequest` it create projection schema of scanner in scanner manager, and serialize the result by the projection. In `HandleContinueScanRequest`, it get scanner with scanner_id from scanner manager, the projection schema is kept in scanner, so it may be irrelevant to the schema kept by tablet_metadata. So I'm wondering it may not have to much impact on deleting old old_schemas_. BTW, it took me some time to thinking about ref-counted approach, as I understand, I considered two approach, 1. Change change schema_ kept by tablet_metadata to shared_ptr wrapped one, and add a lock in SetSchema() and Schema() cause previously atomic swap is broken. It may hurts the perform a lot. 2. Add a atomic counter as Schema field, and decrease ref in scanner dtor, it may also hurt the performance when getting Schema(). > Tablet keeps all history schemas in memory may result in high memory > consumption > > > Key: KUDU-3197 > URL: https://issues.apache.org/jira/browse/KUDU-3197 > Project: Kudu > Issue Type: Improvement > Components: tablet >Affects Versions: 1.12.0 >Reporter: wangningito >Assignee: wangningito >Priority: Minor > Attachments: image-2020-09-25-14-45-33-402.png, > image-2020-09-25-14-49-30-913.png, image-2020-09-25-15-05-44-948.png > > > In case of high frequency of updating table, memory consumption of > kudu-tserver may be very high, and the memory in not tracked in the memory > page. > This is the memory usage of a tablet, the memory consumption of tablet-xxx‘s > peak is 3.6G， but none of its' childrens' memory can reach. > !image-2020-09-25-14-45-33-402.png! > So I use pprof to get the heap sampling. The tserver started for long but the > memory is still consuming by TabletBootstrap:PlayAlterSchemaRequest. > !image-2020-09-25-14-49-30-913.png! > I change the `old_schemas_` in tablet_metadata.h to a fixed size vector, > // Previous values of 'schema_'. > // These are currently kept alive forever, under the assumption that > // a given tablet won't have thousands of "alter table" calls. > // They are kept alive so that callers of schema() don't need to > // worry about reference counting or locking. > std::vector old_schemas_; > The heap sampling then becomes > !image-2020-09-25-15-05-44-948.png! > So, to make application layer more flexible, it could be better to make the > size of the old_schemas configurable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
