Re: Check indexes inline size tool
This one is among other features "missing without a good reason" [1]. [1] https://issues.apache.org/jira/browse/IGNITE-11402 Best regards, Ivan Pavlukhin чт, 30 апр. 2020 г. в 17:18, Ilya Kasnacheev : > > Hello! > > I don't think it ever worked. CREATE INDEX has INLINE_SIZE clause, CREATE > TABLE does not. > > Regards, > -- > Ilya Kasnacheev > > > ср, 29 апр. 2020 г. в 20:35, Denis Magda : > > > Folks, > > > > > > > > > > > Unfortunately and embarrassingly, we still do not support passing > > > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. > > > > > > I'm confused about this statement. Are we talking about an > > issue/regression that slipped into 2.8.0? I do believe the feature worked > > before. > > > > - > > Denis > > > > > > On Tue, Apr 28, 2020 at 4:01 AM Ilya Kasnacheev > > > > wrote: > > > > > Hello! > > > > > > Unfortunately and embarrassingly, we still do not support passing > > > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. > > > This means IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option to create an > > > implicit primary key index with specified inline size. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > вт, 28 апр. 2020 г. в 02:31, Denis Magda : > > > > > > > Hi Sergey, > > > > > > > > Your changes look useful from the application developer perspective. > > > > However, I'm curious why would the one change some low-level > > > > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass > > > > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov < > > > antonovserge...@gmail.com> > > > > wrote: > > > > > > > > > Hi, Igniters! > > > > > > > > > > I'd like to share a new small feature in AI [1]. > > > > > > > > > > For different reasons, the cluster could have a different SQL index > > > > inline > > > > > size [2] on cluster nodes. For example due to > > > > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. > > > > > > > > > > The difference in index inline size may lead to performance > > > degradation. > > > > > I think we must compare inline sizes on node join and warn if > > > difference > > > > > found. Also, We should have the ability to check inline sizes on > > > demand. > > > > > > > > > > I've implemented this check on node join and new command in > > control.sh > > > > > > > > > > Look at warning message and utility command output: > > > > > > > > > > Warn message on a node in the cluster during new node join: > > > > > > > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > > > > > 127.0.0.1:47502 > > > > > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > Inline > > > > > sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > > > > > different. Please drop and create again these indexes to avoid > > > > performance > > > > > problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > > > > > > Warn messages on a joining node, if difference found: > > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > > > 127.0.0.1:47501 > > > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > > Inline sizes on local node and node > > > a86e9cea-63e8-42af-a897-cec4be51 > > > > > are different. Please drop and create again these indexes to avoid > > > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > > > 127.0.0.1:47501 > > > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > > Inline sizes on local node and node > > > a08de16d-df05-48af-a0b9-5596d9c2 > > > > > are different. Please drop and create again these indexes to avoid > > > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) > > > > > > > > > > > > > > > Utility output, if a difference in inline sizes was found: > > > > > > > > > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > > > > > 2020 Copyright(C) Apache Software Foundation > > > > > User: santonov > > > > > Time: 2020-04-27T15:32:25.759 > > > > > Command [CACHE] started > > > > > Arguments: --cache check_index_inline_sizes --yes > > > > > > > > > > > > > > > > > > > > > > > > Found 4 secondary indexes. > > > > > 3 index(es) have different effective inline size on nodes. It can > > lead > > > to > > > > > performance degradation in SQL queries. > > > > > Index(es): > > > > > Full index name:
Re: Check indexes inline size tool
Hello! I don't think it ever worked. CREATE INDEX has INLINE_SIZE clause, CREATE TABLE does not. Regards, -- Ilya Kasnacheev ср, 29 апр. 2020 г. в 20:35, Denis Magda : > Folks, > > > > > > > Unfortunately and embarrassingly, we still do not support passing > > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. > > > I'm confused about this statement. Are we talking about an > issue/regression that slipped into 2.8.0? I do believe the feature worked > before. > > - > Denis > > > On Tue, Apr 28, 2020 at 4:01 AM Ilya Kasnacheev > > wrote: > > > Hello! > > > > Unfortunately and embarrassingly, we still do not support passing > > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. > > This means IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option to create an > > implicit primary key index with specified inline size. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > вт, 28 апр. 2020 г. в 02:31, Denis Magda : > > > > > Hi Sergey, > > > > > > Your changes look useful from the application developer perspective. > > > However, I'm curious why would the one change some low-level > > > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass > > > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. > > > > > > - > > > Denis > > > > > > > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov < > > antonovserge...@gmail.com> > > > wrote: > > > > > > > Hi, Igniters! > > > > > > > > I'd like to share a new small feature in AI [1]. > > > > > > > > For different reasons, the cluster could have a different SQL index > > > inline > > > > size [2] on cluster nodes. For example due to > > > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. > > > > > > > > The difference in index inline size may lead to performance > > degradation. > > > > I think we must compare inline sizes on node join and warn if > > difference > > > > found. Also, We should have the ability to check inline sizes on > > demand. > > > > > > > > I've implemented this check on node join and new command in > control.sh > > > > > > > > Look at warning message and utility command output: > > > > > > > > Warn message on a node in the cluster during new node join: > > > > > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > > > > 127.0.0.1:47502 > > > > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > Inline > > > > sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > > > > different. Please drop and create again these indexes to avoid > > > performance > > > > problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > > > > Warn messages on a joining node, if difference found: > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > > 127.0.0.1:47501 > > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > Inline sizes on local node and node > > a86e9cea-63e8-42af-a897-cec4be51 > > > > are different. Please drop and create again these indexes to avoid > > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > > 127.0.0.1:47501 > > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > Inline sizes on local node and node > > a08de16d-df05-48af-a0b9-5596d9c2 > > > > are different. Please drop and create again these indexes to avoid > > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) > > > > > > > > > > > > Utility output, if a difference in inline sizes was found: > > > > > > > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > > > > 2020 Copyright(C) Apache Software Foundation > > > > User: santonov > > > > Time: 2020-04-27T15:32:25.759 > > > > Command [CACHE] started > > > > Arguments: --cache check_index_inline_sizes --yes > > > > > > > > > > > > > > > > > > Found 4 secondary indexes. > > > > 3 index(es) have different effective inline size on nodes. It can > lead > > to > > > > performance degradation in SQL queries. > > > > Index(es): > > > > Full index name: PUBLIC#TEST_TABLE#L_IDX nodes: > > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > > Full index name: PUBLIC#TEST_TABLE#S1_IDX nodes: > > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > > Full index name: PUBLIC#TEST_TABLE#I_IDX nodes: > > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > >
Re: Check indexes inline size tool
Folks, > > > Unfortunately and embarrassingly, we still do not support passing > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. I'm confused about this statement. Are we talking about an issue/regression that slipped into 2.8.0? I do believe the feature worked before. - Denis On Tue, Apr 28, 2020 at 4:01 AM Ilya Kasnacheev wrote: > Hello! > > Unfortunately and embarrassingly, we still do not support passing > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. > This means IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option to create an > implicit primary key index with specified inline size. > > Regards, > -- > Ilya Kasnacheev > > > вт, 28 апр. 2020 г. в 02:31, Denis Magda : > > > Hi Sergey, > > > > Your changes look useful from the application developer perspective. > > However, I'm curious why would the one change some low-level > > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass > > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. > > > > - > > Denis > > > > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov < > antonovserge...@gmail.com> > > wrote: > > > > > Hi, Igniters! > > > > > > I'd like to share a new small feature in AI [1]. > > > > > > For different reasons, the cluster could have a different SQL index > > inline > > > size [2] on cluster nodes. For example due to > > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. > > > > > > The difference in index inline size may lead to performance > degradation. > > > I think we must compare inline sizes on node join and warn if > difference > > > found. Also, We should have the ability to check inline sizes on > demand. > > > > > > I've implemented this check on node join and new command in control.sh > > > > > > Look at warning message and utility command output: > > > > > > Warn message on a node in the cluster during new node join: > > > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > > > 127.0.0.1:47502 > > > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > Inline > > > sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > > > different. Please drop and create again these indexes to avoid > > performance > > > problems with SQL queries. Problem indexes: > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > > Warn messages on a joining node, if difference found: > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > 127.0.0.1:47501 > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > Inline sizes on local node and node > a86e9cea-63e8-42af-a897-cec4be51 > > > are different. Please drop and create again these indexes to avoid > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > 127.0.0.1:47501 > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > Inline sizes on local node and node > a08de16d-df05-48af-a0b9-5596d9c2 > > > are different. Please drop and create again these indexes to avoid > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) > > > > > > > > > Utility output, if a difference in inline sizes was found: > > > > > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > > > 2020 Copyright(C) Apache Software Foundation > > > User: santonov > > > Time: 2020-04-27T15:32:25.759 > > > Command [CACHE] started > > > Arguments: --cache check_index_inline_sizes --yes > > > > > > > > > > > > Found 4 secondary indexes. > > > 3 index(es) have different effective inline size on nodes. It can lead > to > > > performance degradation in SQL queries. > > > Index(es): > > > Full index name: PUBLIC#TEST_TABLE#L_IDX nodes: > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > Full index name: PUBLIC#TEST_TABLE#S1_IDX nodes: > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > Full index name: PUBLIC#TEST_TABLE#I_IDX nodes: > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > > > > Recommendations: > > > Check that value of property IGNITE_MAX_INDEX_PAYLOAD_SIZE are the > same > > > on all nodes. > > > Recreate indexes (execute DROP INDEX, CREATE INDEX commands) with > > > different inline size. > > > Command [CACHE] finished with code: 0 > > > Control utility has completed execution at: 2020-04-27T15:32:28.025 > > > Execution time: 2266 ms > > > > > > Utility output,
Re: Check indexes inline size tool
Hi, the ticket is ready for review. [1] https://github.com/apache/ignite/pull/7728 вт, 28 апр. 2020 г. в 14:39, Sergey Antonov : > Maxim, I'm talking about cluster upgrade through cluster stop -> binary > update -> cluster start. > > вт, 28 апр. 2020 г. в 14:37, Maxim Muzafarov : > >> Sergey, >> >> Are you talking about a cluster rolling upgrade feature? AFAIK, Apache >> Ignite doesn't support this feature, so why we should care about it? >> >> On Tue, 28 Apr 2020 at 14:32, Sergey Antonov >> wrote: >> > >> > Maxim, >> > >> > > should we _reject_ joining nodes which have different >> > From my point of view, it's a breaking change on cluster update. >> > >> > We can get a different inline size in other scenarios too: as I know we >> did >> > some improvements in calculation effective (actual) index inline size. >> > Let's imagine, we have PDS cluster created on the "old" apache ignite >> > version. We decided to upgrade Ignite version and after that, join a new >> > node to the cluster. On a new node, effective inline sizes will be >> > calculated by the optimized algorithm. On the old nodes, the inline size >> > will not be recalculated. It can lead to a difference in inline sizes >> > without IGNITE_MAX_INDEX_PAYLOAD_SIZE option. >> > >> > >> > >> > вт, 28 апр. 2020 г. в 14:22, Maxim Muzafarov : >> > >> > > Sergey, Ilya, >> > > >> > > >> > > Since inline size for the `create table` clause not supported yet and >> > > the IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option, should we >> > > _reject_ joining nodes which have different >> > > IGNITE_MAX_INDEX_PAYLOAD_SIZE value instead for allowing and printing >> > > warning message? Thus we will force users to have the same values of >> > > IGNITE_MAX_INDEX_PAYLOAD_SIZE property. >> > > >> > > >> > > On Tue, 28 Apr 2020 at 14:01, Ilya Kasnacheev < >> ilya.kasnach...@gmail.com> >> > > wrote: >> > > > >> > > > Hello! >> > > > >> > > > Unfortunately and embarrassingly, we still do not support passing >> > > > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. >> > > > This means IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option to >> create an >> > > > implicit primary key index with specified inline size. >> > > > >> > > > Regards, >> > > > -- >> > > > Ilya Kasnacheev >> > > > >> > > > >> > > > вт, 28 апр. 2020 г. в 02:31, Denis Magda : >> > > > >> > > > > Hi Sergey, >> > > > > >> > > > > Your changes look useful from the application developer >> perspective. >> > > > > However, I'm curious why would the one change some low-level >> > > > > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass >> > > > > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. >> > > > > >> > > > > - >> > > > > Denis >> > > > > >> > > > > >> > > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov < >> > > antonovserge...@gmail.com> >> > > > > wrote: >> > > > > >> > > > > > Hi, Igniters! >> > > > > > >> > > > > > I'd like to share a new small feature in AI [1]. >> > > > > > >> > > > > > For different reasons, the cluster could have a different SQL >> index >> > > > > inline >> > > > > > size [2] on cluster nodes. For example due to >> > > > > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster >> nodes. >> > > > > > >> > > > > > The difference in index inline size may lead to performance >> > > degradation. >> > > > > > I think we must compare inline sizes on node join and warn if >> > > difference >> > > > > > found. Also, We should have the ability to check inline sizes on >> > > demand. >> > > > > > >> > > > > > I've implemented this check on node join and new command in >> > > control.sh >> > > > > > >> > > > > > Look at warning message and utility command output: >> > > > > > >> > > > > > Warn message on a node in the cluster during new node join: >> > > > > > >> > > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 >> > > > > > 127.0.0.1:47502 >> > > > > > >> crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] >> > > > > Inline >> > > > > > sizes on local node and node >> 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are >> > > > > > different. Please drop and create again these indexes to avoid >> > > > > performance >> > > > > > problems with SQL queries. Problem indexes: >> > > > > > >> > > > > > >> > > > > >> > > >> PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) >> > > > > > >> > > > > > Warn messages on a joining node, if difference found: >> > > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea >> > > > > > 127.0.0.1:47501 >> > > > > > >> ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] >> > > > > > Inline sizes on local node and node >> > > a86e9cea-63e8-42af-a897-cec4be51 >> > > > > > are different. Please drop and create again these indexes to >> avoid >> > > > > > performance problems with SQL queries. Problem indexes: >> > > > > > >> > > > > > >> > > > > >> > > >>
Re: Check indexes inline size tool
Maxim, I'm talking about cluster upgrade through cluster stop -> binary update -> cluster start. вт, 28 апр. 2020 г. в 14:37, Maxim Muzafarov : > Sergey, > > Are you talking about a cluster rolling upgrade feature? AFAIK, Apache > Ignite doesn't support this feature, so why we should care about it? > > On Tue, 28 Apr 2020 at 14:32, Sergey Antonov > wrote: > > > > Maxim, > > > > > should we _reject_ joining nodes which have different > > From my point of view, it's a breaking change on cluster update. > > > > We can get a different inline size in other scenarios too: as I know we > did > > some improvements in calculation effective (actual) index inline size. > > Let's imagine, we have PDS cluster created on the "old" apache ignite > > version. We decided to upgrade Ignite version and after that, join a new > > node to the cluster. On a new node, effective inline sizes will be > > calculated by the optimized algorithm. On the old nodes, the inline size > > will not be recalculated. It can lead to a difference in inline sizes > > without IGNITE_MAX_INDEX_PAYLOAD_SIZE option. > > > > > > > > вт, 28 апр. 2020 г. в 14:22, Maxim Muzafarov : > > > > > Sergey, Ilya, > > > > > > > > > Since inline size for the `create table` clause not supported yet and > > > the IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option, should we > > > _reject_ joining nodes which have different > > > IGNITE_MAX_INDEX_PAYLOAD_SIZE value instead for allowing and printing > > > warning message? Thus we will force users to have the same values of > > > IGNITE_MAX_INDEX_PAYLOAD_SIZE property. > > > > > > > > > On Tue, 28 Apr 2020 at 14:01, Ilya Kasnacheev < > ilya.kasnach...@gmail.com> > > > wrote: > > > > > > > > Hello! > > > > > > > > Unfortunately and embarrassingly, we still do not support passing > > > > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. > > > > This means IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option to > create an > > > > implicit primary key index with specified inline size. > > > > > > > > Regards, > > > > -- > > > > Ilya Kasnacheev > > > > > > > > > > > > вт, 28 апр. 2020 г. в 02:31, Denis Magda : > > > > > > > > > Hi Sergey, > > > > > > > > > > Your changes look useful from the application developer > perspective. > > > > > However, I'm curious why would the one change some low-level > > > > > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass > > > > > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov < > > > antonovserge...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi, Igniters! > > > > > > > > > > > > I'd like to share a new small feature in AI [1]. > > > > > > > > > > > > For different reasons, the cluster could have a different SQL > index > > > > > inline > > > > > > size [2] on cluster nodes. For example due to > > > > > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster > nodes. > > > > > > > > > > > > The difference in index inline size may lead to performance > > > degradation. > > > > > > I think we must compare inline sizes on node join and warn if > > > difference > > > > > > found. Also, We should have the ability to check inline sizes on > > > demand. > > > > > > > > > > > > I've implemented this check on node join and new command in > > > control.sh > > > > > > > > > > > > Look at warning message and utility command output: > > > > > > > > > > > > Warn message on a node in the cluster during new node join: > > > > > > > > > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > > > > > > 127.0.0.1:47502 > > > > > > > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > > Inline > > > > > > sizes on local node and node > 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > > > > > > different. Please drop and create again these indexes to avoid > > > > > performance > > > > > > problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > > > > > > > > Warn messages on a joining node, if difference found: > > > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > > > > 127.0.0.1:47501 > > > > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > > > Inline sizes on local node and node > > > a86e9cea-63e8-42af-a897-cec4be51 > > > > > > are different. Please drop and create again these indexes to > avoid > > > > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > > > > 127.0.0.1:47501 > > > > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > > > Inline sizes on local node and node > > >
Re: Check indexes inline size tool
Sergey, Are you talking about a cluster rolling upgrade feature? AFAIK, Apache Ignite doesn't support this feature, so why we should care about it? On Tue, 28 Apr 2020 at 14:32, Sergey Antonov wrote: > > Maxim, > > > should we _reject_ joining nodes which have different > From my point of view, it's a breaking change on cluster update. > > We can get a different inline size in other scenarios too: as I know we did > some improvements in calculation effective (actual) index inline size. > Let's imagine, we have PDS cluster created on the "old" apache ignite > version. We decided to upgrade Ignite version and after that, join a new > node to the cluster. On a new node, effective inline sizes will be > calculated by the optimized algorithm. On the old nodes, the inline size > will not be recalculated. It can lead to a difference in inline sizes > without IGNITE_MAX_INDEX_PAYLOAD_SIZE option. > > > > вт, 28 апр. 2020 г. в 14:22, Maxim Muzafarov : > > > Sergey, Ilya, > > > > > > Since inline size for the `create table` clause not supported yet and > > the IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option, should we > > _reject_ joining nodes which have different > > IGNITE_MAX_INDEX_PAYLOAD_SIZE value instead for allowing and printing > > warning message? Thus we will force users to have the same values of > > IGNITE_MAX_INDEX_PAYLOAD_SIZE property. > > > > > > On Tue, 28 Apr 2020 at 14:01, Ilya Kasnacheev > > wrote: > > > > > > Hello! > > > > > > Unfortunately and embarrassingly, we still do not support passing > > > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. > > > This means IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option to create an > > > implicit primary key index with specified inline size. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > вт, 28 апр. 2020 г. в 02:31, Denis Magda : > > > > > > > Hi Sergey, > > > > > > > > Your changes look useful from the application developer perspective. > > > > However, I'm curious why would the one change some low-level > > > > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass > > > > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov < > > antonovserge...@gmail.com> > > > > wrote: > > > > > > > > > Hi, Igniters! > > > > > > > > > > I'd like to share a new small feature in AI [1]. > > > > > > > > > > For different reasons, the cluster could have a different SQL index > > > > inline > > > > > size [2] on cluster nodes. For example due to > > > > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. > > > > > > > > > > The difference in index inline size may lead to performance > > degradation. > > > > > I think we must compare inline sizes on node join and warn if > > difference > > > > > found. Also, We should have the ability to check inline sizes on > > demand. > > > > > > > > > > I've implemented this check on node join and new command in > > control.sh > > > > > > > > > > Look at warning message and utility command output: > > > > > > > > > > Warn message on a node in the cluster during new node join: > > > > > > > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > > > > > 127.0.0.1:47502 > > > > > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > Inline > > > > > sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > > > > > different. Please drop and create again these indexes to avoid > > > > performance > > > > > problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > > > > > > Warn messages on a joining node, if difference found: > > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > > > 127.0.0.1:47501 > > > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > > Inline sizes on local node and node > > a86e9cea-63e8-42af-a897-cec4be51 > > > > > are different. Please drop and create again these indexes to avoid > > > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > > > 127.0.0.1:47501 > > > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > > Inline sizes on local node and node > > a08de16d-df05-48af-a0b9-5596d9c2 > > > > > are different. Please drop and create again these indexes to avoid > > > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) > > > > > > > > > > > > > > > Utility output, if a difference in inline sizes was found: > > > > > > > > > > Control utility [ver.
Re: Check indexes inline size tool
Maxim, > should we _reject_ joining nodes which have different >From my point of view, it's a breaking change on cluster update. We can get a different inline size in other scenarios too: as I know we did some improvements in calculation effective (actual) index inline size. Let's imagine, we have PDS cluster created on the "old" apache ignite version. We decided to upgrade Ignite version and after that, join a new node to the cluster. On a new node, effective inline sizes will be calculated by the optimized algorithm. On the old nodes, the inline size will not be recalculated. It can lead to a difference in inline sizes without IGNITE_MAX_INDEX_PAYLOAD_SIZE option. вт, 28 апр. 2020 г. в 14:22, Maxim Muzafarov : > Sergey, Ilya, > > > Since inline size for the `create table` clause not supported yet and > the IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option, should we > _reject_ joining nodes which have different > IGNITE_MAX_INDEX_PAYLOAD_SIZE value instead for allowing and printing > warning message? Thus we will force users to have the same values of > IGNITE_MAX_INDEX_PAYLOAD_SIZE property. > > > On Tue, 28 Apr 2020 at 14:01, Ilya Kasnacheev > wrote: > > > > Hello! > > > > Unfortunately and embarrassingly, we still do not support passing > > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. > > This means IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option to create an > > implicit primary key index with specified inline size. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > вт, 28 апр. 2020 г. в 02:31, Denis Magda : > > > > > Hi Sergey, > > > > > > Your changes look useful from the application developer perspective. > > > However, I'm curious why would the one change some low-level > > > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass > > > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. > > > > > > - > > > Denis > > > > > > > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov < > antonovserge...@gmail.com> > > > wrote: > > > > > > > Hi, Igniters! > > > > > > > > I'd like to share a new small feature in AI [1]. > > > > > > > > For different reasons, the cluster could have a different SQL index > > > inline > > > > size [2] on cluster nodes. For example due to > > > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. > > > > > > > > The difference in index inline size may lead to performance > degradation. > > > > I think we must compare inline sizes on node join and warn if > difference > > > > found. Also, We should have the ability to check inline sizes on > demand. > > > > > > > > I've implemented this check on node join and new command in > control.sh > > > > > > > > Look at warning message and utility command output: > > > > > > > > Warn message on a node in the cluster during new node join: > > > > > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > > > > 127.0.0.1:47502 > > > > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > Inline > > > > sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > > > > different. Please drop and create again these indexes to avoid > > > performance > > > > problems with SQL queries. Problem indexes: > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > > > > Warn messages on a joining node, if difference found: > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > > 127.0.0.1:47501 > > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > Inline sizes on local node and node > a86e9cea-63e8-42af-a897-cec4be51 > > > > are different. Please drop and create again these indexes to avoid > > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > > 127.0.0.1:47501 > > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > > Inline sizes on local node and node > a08de16d-df05-48af-a0b9-5596d9c2 > > > > are different. Please drop and create again these indexes to avoid > > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) > > > > > > > > > > > > Utility output, if a difference in inline sizes was found: > > > > > > > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > > > > 2020 Copyright(C) Apache Software Foundation > > > > User: santonov > > > > Time: 2020-04-27T15:32:25.759 > > > > Command [CACHE] started > > > > Arguments: --cache check_index_inline_sizes --yes > > > > > > > > > > > > > > > > Found 4 secondary indexes. > > > > 3 index(es) have different effective inline size on nodes. It can > lead
Re: Check indexes inline size tool
Hello! Unfortunately, that's true. But the user can restart cluster after tables creation and create secondary indexes (CREATE INDEX) after restart. My workaround has a lot of limitations: it doesn't work with in-memory clusters, it's unuseful. вт, 28 апр. 2020 г. в 14:01, Ilya Kasnacheev : > Hello! > > Unfortunately and embarrassingly, we still do not support passing > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. > This means IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option to create an > implicit primary key index with specified inline size. > > Regards, > -- > Ilya Kasnacheev > > > вт, 28 апр. 2020 г. в 02:31, Denis Magda : > > > Hi Sergey, > > > > Your changes look useful from the application developer perspective. > > However, I'm curious why would the one change some low-level > > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass > > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. > > > > - > > Denis > > > > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov < > antonovserge...@gmail.com> > > wrote: > > > > > Hi, Igniters! > > > > > > I'd like to share a new small feature in AI [1]. > > > > > > For different reasons, the cluster could have a different SQL index > > inline > > > size [2] on cluster nodes. For example due to > > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. > > > > > > The difference in index inline size may lead to performance > degradation. > > > I think we must compare inline sizes on node join and warn if > difference > > > found. Also, We should have the ability to check inline sizes on > demand. > > > > > > I've implemented this check on node join and new command in control.sh > > > > > > Look at warning message and utility command output: > > > > > > Warn message on a node in the cluster during new node join: > > > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > > > 127.0.0.1:47502 > > > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > Inline > > > sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > > > different. Please drop and create again these indexes to avoid > > performance > > > problems with SQL queries. Problem indexes: > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > > Warn messages on a joining node, if difference found: > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > 127.0.0.1:47501 > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > Inline sizes on local node and node > a86e9cea-63e8-42af-a897-cec4be51 > > > are different. Please drop and create again these indexes to avoid > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > 127.0.0.1:47501 > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > Inline sizes on local node and node > a08de16d-df05-48af-a0b9-5596d9c2 > > > are different. Please drop and create again these indexes to avoid > > > performance problems with SQL queries. Problem indexes: > > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) > > > > > > > > > Utility output, if a difference in inline sizes was found: > > > > > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > > > 2020 Copyright(C) Apache Software Foundation > > > User: santonov > > > Time: 2020-04-27T15:32:25.759 > > > Command [CACHE] started > > > Arguments: --cache check_index_inline_sizes --yes > > > > > > > > > > > > Found 4 secondary indexes. > > > 3 index(es) have different effective inline size on nodes. It can lead > to > > > performance degradation in SQL queries. > > > Index(es): > > > Full index name: PUBLIC#TEST_TABLE#L_IDX nodes: > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > Full index name: PUBLIC#TEST_TABLE#S1_IDX nodes: > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > Full index name: PUBLIC#TEST_TABLE#I_IDX nodes: > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > > > > Recommendations: > > > Check that value of property IGNITE_MAX_INDEX_PAYLOAD_SIZE are the > same > > > on all nodes. > > > Recreate indexes (execute DROP INDEX, CREATE INDEX commands) with > > > different inline size. > > > Command [CACHE] finished with code: 0 > > > Control utility has completed execution at: 2020-04-27T15:32:28.025 > > > Execution time: 2266 ms > > > > > > Utility output, if all indexes have the same inline size: > > >
Re: Check indexes inline size tool
Sergey, Ilya, Since inline size for the `create table` clause not supported yet and the IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option, should we _reject_ joining nodes which have different IGNITE_MAX_INDEX_PAYLOAD_SIZE value instead for allowing and printing warning message? Thus we will force users to have the same values of IGNITE_MAX_INDEX_PAYLOAD_SIZE property. On Tue, 28 Apr 2020 at 14:01, Ilya Kasnacheev wrote: > > Hello! > > Unfortunately and embarrassingly, we still do not support passing > INLINE_SIZE to CREATE TABLE, at least in 2.8.0. > This means IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option to create an > implicit primary key index with specified inline size. > > Regards, > -- > Ilya Kasnacheev > > > вт, 28 апр. 2020 г. в 02:31, Denis Magda : > > > Hi Sergey, > > > > Your changes look useful from the application developer perspective. > > However, I'm curious why would the one change some low-level > > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass > > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. > > > > - > > Denis > > > > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov > > wrote: > > > > > Hi, Igniters! > > > > > > I'd like to share a new small feature in AI [1]. > > > > > > For different reasons, the cluster could have a different SQL index > > inline > > > size [2] on cluster nodes. For example due to > > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. > > > > > > The difference in index inline size may lead to performance degradation. > > > I think we must compare inline sizes on node join and warn if difference > > > found. Also, We should have the ability to check inline sizes on demand. > > > > > > I've implemented this check on node join and new command in control.sh > > > > > > Look at warning message and utility command output: > > > > > > Warn message on a node in the cluster during new node join: > > > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > > > 127.0.0.1:47502 > > > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > Inline > > > sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > > > different. Please drop and create again these indexes to avoid > > performance > > > problems with SQL queries. Problem indexes: > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > > > Warn messages on a joining node, if difference found: > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > 127.0.0.1:47501 > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > Inline sizes on local node and node a86e9cea-63e8-42af-a897-cec4be51 > > > are different. Please drop and create again these indexes to avoid > > > performance problems with SQL queries. Problem indexes: > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > > 127.0.0.1:47501 > > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > > Inline sizes on local node and node a08de16d-df05-48af-a0b9-5596d9c2 > > > are different. Please drop and create again these indexes to avoid > > > performance problems with SQL queries. Problem indexes: > > > > > > > > PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) > > > > > > > > > Utility output, if a difference in inline sizes was found: > > > > > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > > > 2020 Copyright(C) Apache Software Foundation > > > User: santonov > > > Time: 2020-04-27T15:32:25.759 > > > Command [CACHE] started > > > Arguments: --cache check_index_inline_sizes --yes > > > > > > > > > > > Found 4 secondary indexes. > > > 3 index(es) have different effective inline size on nodes. It can lead to > > > performance degradation in SQL queries. > > > Index(es): > > > Full index name: PUBLIC#TEST_TABLE#L_IDX nodes: > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > Full index name: PUBLIC#TEST_TABLE#S1_IDX nodes: > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > Full index name: PUBLIC#TEST_TABLE#I_IDX nodes: > > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > > > > Recommendations: > > > Check that value of property IGNITE_MAX_INDEX_PAYLOAD_SIZE are the same > > > on all nodes. > > > Recreate indexes (execute DROP INDEX, CREATE INDEX commands) with > > > different inline size. > > > Command [CACHE] finished with code: 0 > > > Control utility has completed execution at: 2020-04-27T15:32:28.025 > > > Execution time:
Re: Check indexes inline size tool
Hello! Unfortunately and embarrassingly, we still do not support passing INLINE_SIZE to CREATE TABLE, at least in 2.8.0. This means IGNITE_MAX_INDEX_PAYLOAD_SIZE is the only option to create an implicit primary key index with specified inline size. Regards, -- Ilya Kasnacheev вт, 28 апр. 2020 г. в 02:31, Denis Magda : > Hi Sergey, > > Your changes look useful from the application developer perspective. > However, I'm curious why would the one change some low-level > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. > > - > Denis > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov > wrote: > > > Hi, Igniters! > > > > I'd like to share a new small feature in AI [1]. > > > > For different reasons, the cluster could have a different SQL index > inline > > size [2] on cluster nodes. For example due to > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. > > > > The difference in index inline size may lead to performance degradation. > > I think we must compare inline sizes on node join and warn if difference > > found. Also, We should have the ability to check inline sizes on demand. > > > > I've implemented this check on node join and new command in control.sh > > > > Look at warning message and utility command output: > > > > Warn message on a node in the cluster during new node join: > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > > 127.0.0.1:47502 > > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > Inline > > sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > > different. Please drop and create again these indexes to avoid > performance > > problems with SQL queries. Problem indexes: > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > Warn messages on a joining node, if difference found: > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > 127.0.0.1:47501 > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > Inline sizes on local node and node a86e9cea-63e8-42af-a897-cec4be51 > > are different. Please drop and create again these indexes to avoid > > performance problems with SQL queries. Problem indexes: > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > 127.0.0.1:47501 > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > Inline sizes on local node and node a08de16d-df05-48af-a0b9-5596d9c2 > > are different. Please drop and create again these indexes to avoid > > performance problems with SQL queries. Problem indexes: > > > > > PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) > > > > > > Utility output, if a difference in inline sizes was found: > > > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > > 2020 Copyright(C) Apache Software Foundation > > User: santonov > > Time: 2020-04-27T15:32:25.759 > > Command [CACHE] started > > Arguments: --cache check_index_inline_sizes --yes > > > > > > > Found 4 secondary indexes. > > 3 index(es) have different effective inline size on nodes. It can lead to > > performance degradation in SQL queries. > > Index(es): > > Full index name: PUBLIC#TEST_TABLE#L_IDX nodes: > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > Full index name: PUBLIC#TEST_TABLE#S1_IDX nodes: > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > Full index name: PUBLIC#TEST_TABLE#I_IDX nodes: > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > > Recommendations: > > Check that value of property IGNITE_MAX_INDEX_PAYLOAD_SIZE are the same > > on all nodes. > > Recreate indexes (execute DROP INDEX, CREATE INDEX commands) with > > different inline size. > > Command [CACHE] finished with code: 0 > > Control utility has completed execution at: 2020-04-27T15:32:28.025 > > Execution time: 2266 ms > > > > Utility output, if all indexes have the same inline size: > > > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > > 2020 Copyright(C) Apache Software Foundation > > User: santonov > > Time: 2020-04-27T15:30:20.950 > > Command [CACHE] started > > Arguments: --cache check_index_inline_sizes --yes > > > > > > > Found 2 secondary indexes. > > All secondary indexes have the same effective inline size on all cluster > > nodes. > > Command [CACHE] finished with code: 0 > > Control utility has completed execution at: 2020-04-27T15:30:23.428 > >
Re: Check indexes inline size tool
Hi Denis, The problem could be shown when you invoke CREATE INDEX without the INLINE_SIZE parameter. You don't face with described problem If index creates by CREATE_INDEX with explicit INLINE SIZE value. вт, 28 апр. 2020 г. в 02:31, Denis Magda : > Hi Sergey, > > Your changes look useful from the application developer perspective. > However, I'm curious why would the one change some low-level > IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass > INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. > > - > Denis > > > On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov > wrote: > > > Hi, Igniters! > > > > I'd like to share a new small feature in AI [1]. > > > > For different reasons, the cluster could have a different SQL index > inline > > size [2] on cluster nodes. For example due to > > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. > > > > The difference in index inline size may lead to performance degradation. > > I think we must compare inline sizes on node join and warn if difference > > found. Also, We should have the ability to check inline sizes on demand. > > > > I've implemented this check on node join and new command in control.sh > > > > Look at warning message and utility command output: > > > > Warn message on a node in the cluster during new node join: > > > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > > 127.0.0.1:47502 > > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > Inline > > sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > > different. Please drop and create again these indexes to avoid > performance > > problems with SQL queries. Problem indexes: > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > > > Warn messages on a joining node, if difference found: > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > 127.0.0.1:47501 > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > Inline sizes on local node and node a86e9cea-63e8-42af-a897-cec4be51 > > are different. Please drop and create again these indexes to avoid > > performance problems with SQL queries. Problem indexes: > > > > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > > 127.0.0.1:47501 > > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > > Inline sizes on local node and node a08de16d-df05-48af-a0b9-5596d9c2 > > are different. Please drop and create again these indexes to avoid > > performance problems with SQL queries. Problem indexes: > > > > > PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) > > > > > > Utility output, if a difference in inline sizes was found: > > > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > > 2020 Copyright(C) Apache Software Foundation > > User: santonov > > Time: 2020-04-27T15:32:25.759 > > Command [CACHE] started > > Arguments: --cache check_index_inline_sizes --yes > > > > > > > Found 4 secondary indexes. > > 3 index(es) have different effective inline size on nodes. It can lead to > > performance degradation in SQL queries. > > Index(es): > > Full index name: PUBLIC#TEST_TABLE#L_IDX nodes: > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > Full index name: PUBLIC#TEST_TABLE#S1_IDX nodes: > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > Full index name: PUBLIC#TEST_TABLE#I_IDX nodes: > > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > > > Recommendations: > > Check that value of property IGNITE_MAX_INDEX_PAYLOAD_SIZE are the same > > on all nodes. > > Recreate indexes (execute DROP INDEX, CREATE INDEX commands) with > > different inline size. > > Command [CACHE] finished with code: 0 > > Control utility has completed execution at: 2020-04-27T15:32:28.025 > > Execution time: 2266 ms > > > > Utility output, if all indexes have the same inline size: > > > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > > 2020 Copyright(C) Apache Software Foundation > > User: santonov > > Time: 2020-04-27T15:30:20.950 > > Command [CACHE] started > > Arguments: --cache check_index_inline_sizes --yes > > > > > > > Found 2 secondary indexes. > > All secondary indexes have the same effective inline size on all cluster > > nodes. > > Command [CACHE] finished with code: 0 > > Control utility has completed execution at: 2020-04-27T15:30:23.428 > > Execution time: 2478 ms > > > > Any objections? > > > > [1]
Re: Check indexes inline size tool
Hi Sergey, Your changes look useful from the application developer perspective. However, I'm curious why would the one change some low-level IGNITE_MAX_INDEX_PAYLOAD_SIZE parameter when it's advised to pass INLINE_SIZE to CREATE TABLE to change the index size cluster-wide. - Denis On Mon, Apr 27, 2020 at 5:38 AM Sergey Antonov wrote: > Hi, Igniters! > > I'd like to share a new small feature in AI [1]. > > For different reasons, the cluster could have a different SQL index inline > size [2] on cluster nodes. For example due to > different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. > > The difference in index inline size may lead to performance degradation. > I think we must compare inline sizes on node join and warn if difference > found. Also, We should have the ability to check inline sizes on demand. > > I've implemented this check on node join and new command in control.sh > > Look at warning message and utility command output: > > Warn message on a node in the cluster during new node join: > > [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 > 127.0.0.1:47502 > crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] Inline > sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are > different. Please drop and create again these indexes to avoid performance > problems with SQL queries. Problem indexes: > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > > Warn messages on a joining node, if difference found: > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > 127.0.0.1:47501 > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > Inline sizes on local node and node a86e9cea-63e8-42af-a897-cec4be51 > are different. Please drop and create again these indexes to avoid > performance problems with SQL queries. Problem indexes: > > PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) > [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea > 127.0.0.1:47501 > ]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] > Inline sizes on local node and node a08de16d-df05-48af-a0b9-5596d9c2 > are different. Please drop and create again these indexes to avoid > performance problems with SQL queries. Problem indexes: > > PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) > > > Utility output, if a difference in inline sizes was found: > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > 2020 Copyright(C) Apache Software Foundation > User: santonov > Time: 2020-04-27T15:32:25.759 > Command [CACHE] started > Arguments: --cache check_index_inline_sizes --yes > > > Found 4 secondary indexes. > 3 index(es) have different effective inline size on nodes. It can lead to > performance degradation in SQL queries. > Index(es): > Full index name: PUBLIC#TEST_TABLE#L_IDX nodes: > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > Full index name: PUBLIC#TEST_TABLE#S1_IDX nodes: > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > Full index name: PUBLIC#TEST_TABLE#I_IDX nodes: > [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: > [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 > > Recommendations: > Check that value of property IGNITE_MAX_INDEX_PAYLOAD_SIZE are the same > on all nodes. > Recreate indexes (execute DROP INDEX, CREATE INDEX commands) with > different inline size. > Command [CACHE] finished with code: 0 > Control utility has completed execution at: 2020-04-27T15:32:28.025 > Execution time: 2266 ms > > Utility output, if all indexes have the same inline size: > > Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] > 2020 Copyright(C) Apache Software Foundation > User: santonov > Time: 2020-04-27T15:30:20.950 > Command [CACHE] started > Arguments: --cache check_index_inline_sizes --yes > > > Found 2 secondary indexes. > All secondary indexes have the same effective inline size on all cluster > nodes. > Command [CACHE] finished with code: 0 > Control utility has completed execution at: 2020-04-27T15:30:23.428 > Execution time: 2478 ms > > Any objections? > > [1] https://issues.apache.org/jira/browse/IGNITE-12942 > [2] https://apacheignite-sql.readme.io/docs/create-index > [3] > > https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteSystemProperties.html#IGNITE_MAX_INDEX_PAYLOAD_SIZE > > -- > BR, Sergey Antonov >
Check indexes inline size tool
Hi, Igniters! I'd like to share a new small feature in AI [1]. For different reasons, the cluster could have a different SQL index inline size [2] on cluster nodes. For example due to different IGNITE_MAX_INDEX_PAYLOAD_SIZE [3] value on cluster nodes. The difference in index inline size may lead to performance degradation. I think we must compare inline sizes on node join and warn if difference found. Also, We should have the ability to check inline sizes on demand. I've implemented this check on node join and new command in control.sh Look at warning message and utility command output: Warn message on a node in the cluster during new node join: [2020-04-27 15:36:21,185][WARN ][tcp-disco-msg-worker-[6ba0b823 127.0.0.1:47502 crd]-#17%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] Inline sizes on local node and node 5bf6ca48-34a0-4aff-8db2-c0b6df303d3f are different. Please drop and create again these indexes to avoid performance problems with SQL queries. Problem indexes: PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) Warn messages on a joining node, if difference found: [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea 127.0.0.1:47501]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] Inline sizes on local node and node a86e9cea-63e8-42af-a897-cec4be51 are different. Please drop and create again these indexes to avoid performance problems with SQL queries. Problem indexes: PUBLIC#TEST_TABLE#L_IDX(1,2),PUBLIC#TEST_TABLE#S1_IDX(1,2),PUBLIC#TEST_TABLE#I_IDX(1,2) [2020-04-27 15:35:17,326][WARN ][tcp-disco-msg-worker-[a86e9cea 127.0.0.1:47501]-#11%cache.CheckIndexesInlineSizeOnNodeJoinMultiJvmTest0%][root] Inline sizes on local node and node a08de16d-df05-48af-a0b9-5596d9c2 are different. Please drop and create again these indexes to avoid performance problems with SQL queries. Problem indexes: PUBLIC#TEST_TABLE#L_IDX(1,3),PUBLIC#TEST_TABLE#S1_IDX(1,3),PUBLIC#TEST_TABLE#I_IDX(1,3) Utility output, if a difference in inline sizes was found: Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] 2020 Copyright(C) Apache Software Foundation User: santonov Time: 2020-04-27T15:32:25.759 Command [CACHE] started Arguments: --cache check_index_inline_sizes --yes Found 4 secondary indexes. 3 index(es) have different effective inline size on nodes. It can lead to performance degradation in SQL queries. Index(es): Full index name: PUBLIC#TEST_TABLE#L_IDX nodes: [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 Full index name: PUBLIC#TEST_TABLE#S1_IDX nodes: [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 Full index name: PUBLIC#TEST_TABLE#I_IDX nodes: [ca1d2bae-89d4-4e8d-ae11-6c68f390] inline size: 1, nodes: [8327abd1-df08-4b97-8720-de95e363e745] inline size: 2 Recommendations: Check that value of property IGNITE_MAX_INDEX_PAYLOAD_SIZE are the same on all nodes. Recreate indexes (execute DROP INDEX, CREATE INDEX commands) with different inline size. Command [CACHE] finished with code: 0 Control utility has completed execution at: 2020-04-27T15:32:28.025 Execution time: 2266 ms Utility output, if all indexes have the same inline size: Control utility [ver. 2.9.0-SNAPSHOT#20200427-sha1:DEV] 2020 Copyright(C) Apache Software Foundation User: santonov Time: 2020-04-27T15:30:20.950 Command [CACHE] started Arguments: --cache check_index_inline_sizes --yes Found 2 secondary indexes. All secondary indexes have the same effective inline size on all cluster nodes. Command [CACHE] finished with code: 0 Control utility has completed execution at: 2020-04-27T15:30:23.428 Execution time: 2478 ms Any objections? [1] https://issues.apache.org/jira/browse/IGNITE-12942 [2] https://apacheignite-sql.readme.io/docs/create-index [3] https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteSystemProperties.html#IGNITE_MAX_INDEX_PAYLOAD_SIZE -- BR, Sergey Antonov