[jira] [Updated] (IGNITE-17484) Tech debt with static context in Question and Cmd providing
[ https://issues.apache.org/jira/browse/IGNITE-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Pochatkin updated IGNITE-17484: --- Component/s: cli > Tech debt with static context in Question and Cmd providing > --- > > Key: IGNITE-17484 > URL: https://issues.apache.org/jira/browse/IGNITE-17484 > Project: Ignite > Issue Type: Improvement > Components: cli >Reporter: Mikhail Pochatkin >Priority: Major > Labels: ignite-3, ignite-3-cli-tool > > Currently, org.apache.ignite.cli.core.flow.question.QuestionAskerFactory and > org.apache.ignite.cli.core.repl.context.CommandLineContextProvider has a > static context which should be refactored in non-static model with better > injection logic. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-16970) IEP-88: CLI Tool
[ https://issues.apache.org/jira/browse/IGNITE-16970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Pochatkin updated IGNITE-16970: --- Component/s: cli > IEP-88: CLI Tool > > > Key: IGNITE-16970 > URL: https://issues.apache.org/jira/browse/IGNITE-16970 > Project: Ignite > Issue Type: Epic > Components: cli >Reporter: Aleksandr >Priority: Major > Labels: ignite-3, ignite-3-cli-tool > > Develop an advanced CLI Tool with an explicit focus on usability. > A user should download the tool from the website and then use it to: > * Connect to a cluster to monitor its state and perform management > operations (configuration changes, cluster init). > * Connect to a cluster to run SQL queries. > * Connect to a cluster to get the cluster status (topology, version). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17102) Implement cluster show command
[ https://issues.apache.org/jira/browse/IGNITE-17102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Pochatkin updated IGNITE-17102: --- Component/s: cli > Implement cluster show command > -- > > Key: IGNITE-17102 > URL: https://issues.apache.org/jira/browse/IGNITE-17102 > Project: Ignite > Issue Type: Task > Components: cli >Reporter: Aleksandr >Priority: Major > Labels: ignite-3, ignite-3-cli-tool > > Now `cluster show` is an alias for `status` but according to the IEP it > should display wider information. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17170) Drop deprecated package
[ https://issues.apache.org/jira/browse/IGNITE-17170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Pochatkin updated IGNITE-17170: --- Component/s: cli > Drop deprecated package > > > Key: IGNITE-17170 > URL: https://issues.apache.org/jira/browse/IGNITE-17170 > Project: Ignite > Issue Type: Improvement > Components: cli >Reporter: Mikhail Pochatkin >Priority: Major > Labels: ignite-3, ignite-3-cli-tool > > After migrate to new CLI (IGNITE-16971) to support backward compatibility, > existed commands (cluster/node bootstrap, start, stop ...) moved to > deprecated module. Need to rewrite this commands to new Pipeline architecture > and drop all not needed code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17569) Add an option to display plain tables
[ https://issues.apache.org/jira/browse/IGNITE-17569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Pochatkin updated IGNITE-17569: --- Epic Link: IGNITE-16970 > Add an option to display plain tables > - > > Key: IGNITE-17569 > URL: https://issues.apache.org/jira/browse/IGNITE-17569 > Project: Ignite > Issue Type: Task > Components: cli >Reporter: Aleksandr >Priority: Major > Labels: ignite-3, ignite-3-cli-tool > > Now CLI shows tables that are pretty rendered. This looks grate but makes > hard to apply in any pipeline. For example, writing an awk script to show all > node names from the cluster might be a challange. > I suggest to add an option (--plain or something) that will make the CLI to > display tables with the plain forrmatting. It coud be columns splitted with > \t. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17778) C++ 3.0: Fix licenses in main
[ https://issues.apache.org/jira/browse/IGNITE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610789#comment-17610789 ] Pavel Tupitsyn commented on IGNITE-17778: - [~isapego] looks good. > C++ 3.0: Fix licenses in main > - > > Key: IGNITE-17778 > URL: https://issues.apache.org/jira/browse/IGNITE-17778 > Project: Ignite > Issue Type: Improvement > Components: platforms >Reporter: Igor Sapego >Assignee: Igor Sapego >Priority: Major > Labels: ignite-3 > > License check fails in main: > https://ci.ignite.apache.org/buildConfiguration/ignite3_Test_SanityChecks_LicensesHeaders/6793837 > Files with unapproved licenses: > modules/platforms/cpp/conanfile.txt > modules/platforms/cpp/common/guid.h -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17775) Invalid data in network buffers causes message deserialization errors and messages loss
[
https://issues.apache.org/jira/browse/IGNITE-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-17775:
Component/s: networking
> Invalid data in network buffers causes message deserialization errors and
> messages loss
> ---
>
> Key: IGNITE-17775
> URL: https://issues.apache.org/jira/browse/IGNITE-17775
> Project: Ignite
> Issue Type: Bug
> Components: networking
>Reporter: Denis Chudov
>Priority: Major
> Labels: ignite-3
>
> In some tests I observe network messages' deserialization errors and timeout
> exceptions while waiting for response. In some cases there is negative group
> type of the message, and this causes error:
> {code:java}
> java.lang.AssertionError: message type must not be negative, messageType=-5376
> at
> org.apache.ignite.network.MessageSerializationRegistryImpl.getFactory(MessageSerializationRegistryImpl.java:77)
> at
> org.apache.ignite.network.MessageSerializationRegistryImpl.createDeserializer(MessageSerializationRegistryImpl.java:102)
> at
> org.apache.ignite.internal.network.serialization.SerializationService.createDeserializer(SerializationService.java:68)
> at
> org.apache.ignite.internal.network.serialization.PerSessionSerializationService.createMessageDeserializer(PerSessionSerializationService.java:109)
> at
> org.apache.ignite.internal.network.netty.InboundDecoder.decode(InboundDecoder.java:89)
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:833)
> {code}
> When the group or message type is positive but not existing, there should be
> a NetworkConfigurationException but it's not displayed in logs, however, it
> causes TimeoutExceptions because of messages loss.
> This reproduces in
> https://github.com/gridgain/apache-ignite-3/tree/ignite-17523-2 in
> ItTablesApiTest#testGetTableFromLaggedNode
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-17778) C++ 3.0: Fix licenses in main
[ https://issues.apache.org/jira/browse/IGNITE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Sapego reassigned IGNITE-17778: Assignee: Igor Sapego > C++ 3.0: Fix licenses in main > - > > Key: IGNITE-17778 > URL: https://issues.apache.org/jira/browse/IGNITE-17778 > Project: Ignite > Issue Type: Improvement > Components: platforms >Reporter: Igor Sapego >Assignee: Igor Sapego >Priority: Major > Labels: ignite-3 > > License check fails in main: > https://ci.ignite.apache.org/buildConfiguration/ignite3_Test_SanityChecks_LicensesHeaders/6793837 > Files with unapproved licenses: > modules/platforms/cpp/conanfile.txt > modules/platforms/cpp/common/guid.h -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17778) C++ 3.0: Fix licenses in main
Igor Sapego created IGNITE-17778: Summary: C++ 3.0: Fix licenses in main Key: IGNITE-17778 URL: https://issues.apache.org/jira/browse/IGNITE-17778 Project: Ignite Issue Type: Improvement Components: platforms Reporter: Igor Sapego License check fails in main: https://ci.ignite.apache.org/buildConfiguration/ignite3_Test_SanityChecks_LicensesHeaders/6793837 Files with unapproved licenses: modules/platforms/cpp/conanfile.txt modules/platforms/cpp/common/guid.h -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17777) Thin 3.0: use BinaryTuple for Compute and SQL results and arguments
[ https://issues.apache.org/jira/browse/IGNITE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn updated IGNITE-1: Description: Thin client protocol currently uses MsgPack types to serialize Compute arguments and results, SQL arguments and results. This is not consistent with Table API and complicates the protocol. All user data in thin client protocol should be serialized in BinaryTuple format. > Thin 3.0: use BinaryTuple for Compute and SQL results and arguments > --- > > Key: IGNITE-1 > URL: https://issues.apache.org/jira/browse/IGNITE-1 > Project: Ignite > Issue Type: New Feature > Components: thin client >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > > Thin client protocol currently uses MsgPack types to serialize Compute > arguments and results, SQL arguments and results. > This is not consistent with Table API and complicates the protocol. > All user data in thin client protocol should be serialized in BinaryTuple > format. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-16510) Calcite engine. Support "keep binary" flag
[ https://issues.apache.org/jira/browse/IGNITE-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Plekhanov updated IGNITE-16510: --- Labels: calcite calcite3-required (was: calcite calcite2-required calcite3-required) > Calcite engine. Support "keep binary" flag > -- > > Key: IGNITE-16510 > URL: https://issues.apache.org/jira/browse/IGNITE-16510 > Project: Ignite > Issue Type: New Feature >Reporter: Aleksey Plekhanov >Assignee: Aleksey Plekhanov >Priority: Major > Labels: calcite, calcite3-required > Fix For: 2.15 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The "keep binary" flag is currently ignored by the Calcite-based SQL engine > and there is no way to return a deserialized object: all POJO returned in > binary format. We should support the "keep binary" flag and deserialize > objects if needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17777) Thin 3.0: use BinaryTuple for Compute and SQL results and arguments
[ https://issues.apache.org/jira/browse/IGNITE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn updated IGNITE-1: Summary: Thin 3.0: use BinaryTuple for Compute and SQL results and arguments (was: Thin 3.0: use BinaryTuple for Compute and SQL results and argument) > Thin 3.0: use BinaryTuple for Compute and SQL results and arguments > --- > > Key: IGNITE-1 > URL: https://issues.apache.org/jira/browse/IGNITE-1 > Project: Ignite > Issue Type: New Feature > Components: thin client >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17777) Thin 3.0: use BinaryTuple for Compute and SQL results and argument
Pavel Tupitsyn created IGNITE-1: --- Summary: Thin 3.0: use BinaryTuple for Compute and SQL results and argument Key: IGNITE-1 URL: https://issues.apache.org/jira/browse/IGNITE-1 Project: Ignite Issue Type: New Feature Components: thin client Reporter: Pavel Tupitsyn Assignee: Pavel Tupitsyn Fix For: 3.0.0-alpha6 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-16510) Calcite engine. Support "keep binary" flag
[
https://issues.apache.org/jira/browse/IGNITE-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610672#comment-17610672
]
Ignite TC Bot commented on IGNITE-16510:
{panel:title=Branch: [pull/10272/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/10272/head] Base: [master] : New Tests
(1)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Calcite SQL{color} [[tests
1|https://ci2.ignite.apache.org/viewLog.html?buildId=6631954]]
* {color:#013220}IgniteCalciteTestSuite:
KeepBinaryIntegrationTest.testKeepBinary - PASSED{color}
{panel}
[TeamCity *-- Run :: All*
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=6631293buildTypeId=IgniteTests24Java8_RunAll]
> Calcite engine. Support "keep binary" flag
> --
>
> Key: IGNITE-16510
> URL: https://issues.apache.org/jira/browse/IGNITE-16510
> Project: Ignite
> Issue Type: New Feature
>Reporter: Aleksey Plekhanov
>Assignee: Aleksey Plekhanov
>Priority: Major
> Labels: calcite, calcite2-required, calcite3-required
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The "keep binary" flag is currently ignored by the Calcite-based SQL engine
> and there is no way to return a deserialized object: all POJO returned in
> binary format. We should support the "keep binary" flag and deserialize
> objects if needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17717) Logging cdc in ignite2ignite mode
[ https://issues.apache.org/jira/browse/IGNITE-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Izhikov updated IGNITE-17717: - Attachment: 3b799724-998a-434b-8ca3-eb9877490ce9.png > Logging cdc in ignite2ignite mode > - > > Key: IGNITE-17717 > URL: https://issues.apache.org/jira/browse/IGNITE-17717 > Project: Ignite > Issue Type: Task >Reporter: Luchnikov Alexander >Assignee: Nikolay Izhikov >Priority: Major > Labels: ise > Attachments: 3b799724-998a-434b-8ca3-eb9877490ce9.png > > > When using cdc in ignite2ignite mode, there is a problem with logging. > When running ignite-cdc.sh, the log is written to ignite-cdc.log until the > ignite client starts, after it starts, the recording continues to ignite.log. > Probably the problem is related to the replacement of appId at the start of > the client node. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17717) Logging cdc in ignite2ignite mode
[ https://issues.apache.org/jira/browse/IGNITE-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610644#comment-17610644 ] Nikolay Izhikov commented on IGNITE-17717: -- [~AldoRaine] default logging properties leads to correct behaviour. Can you, please, clarify configuration that leads to exception. !3b799724-998a-434b-8ca3-eb9877490ce9.png! > Logging cdc in ignite2ignite mode > - > > Key: IGNITE-17717 > URL: https://issues.apache.org/jira/browse/IGNITE-17717 > Project: Ignite > Issue Type: Task >Reporter: Luchnikov Alexander >Assignee: Nikolay Izhikov >Priority: Major > Labels: ise > Attachments: 3b799724-998a-434b-8ca3-eb9877490ce9.png > > > When using cdc in ignite2ignite mode, there is a problem with logging. > When running ignite-cdc.sh, the log is written to ignite-cdc.log until the > ignite client starts, after it starts, the recording continues to ignite.log. > Probably the problem is related to the replacement of appId at the start of > the client node. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17743) Expose datastreamer receivers/updaters.
[ https://issues.apache.org/jira/browse/IGNITE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Steshin updated IGNITE-17743: -- Description: There are several prepared streamer receivers: batched, batchedsorted, individual, isolated. Batched could be used instead of individual for atomic caches whe allowOverwrite is set ti True. Datastreamer allows user to choose one. But there is no docs, examples. Also, allowOverwrite option depends on the receiver. It should be moved or be bound to receiver. was: There are several prepared streamer receivers: batched, batchedsorted, individual, isolated. Datastreamer allows user to choose one. But there is no docs, examples. Also, allowOverwrite option depends on the receiver. It should be moved or be bound to receiver. > Expose datastreamer receivers/updaters. > --- > > Key: IGNITE-17743 > URL: https://issues.apache.org/jira/browse/IGNITE-17743 > Project: Ignite > Issue Type: Sub-task >Reporter: Vladimir Steshin >Assignee: Vladimir Steshin >Priority: Minor > Labels: ise > > There are several prepared streamer receivers: batched, batchedsorted, > individual, isolated. > Batched could be used instead of individual for atomic caches whe > allowOverwrite is set ti True. > Datastreamer allows user to choose one. But there is no docs, examples. > Also, allowOverwrite option depends on the receiver. It should be moved or be > bound to receiver. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17743) Expose datastreamer receivers/updaters.
[ https://issues.apache.org/jira/browse/IGNITE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Steshin updated IGNITE-17743: -- Description: There are several prepared streamer receivers: batched, batchedsorted, individual, isolated. Batched could be used instead of individual for atomic caches whe allowOverwrite is set to True. Datastreamer allows user to choose one. But there is no docs, examples. Also, allowOverwrite option depends on the receiver. It should be moved or be bound to receiver. was: There are several prepared streamer receivers: batched, batchedsorted, individual, isolated. Batched could be used instead of individual for atomic caches whe allowOverwrite is set ti True. Datastreamer allows user to choose one. But there is no docs, examples. Also, allowOverwrite option depends on the receiver. It should be moved or be bound to receiver. > Expose datastreamer receivers/updaters. > --- > > Key: IGNITE-17743 > URL: https://issues.apache.org/jira/browse/IGNITE-17743 > Project: Ignite > Issue Type: Sub-task >Reporter: Vladimir Steshin >Assignee: Vladimir Steshin >Priority: Minor > Labels: ise > > There are several prepared streamer receivers: batched, batchedsorted, > individual, isolated. > Batched could be used instead of individual for atomic caches whe > allowOverwrite is set to True. > Datastreamer allows user to choose one. But there is no docs, examples. > Also, allowOverwrite option depends on the receiver. It should be moved or be > bound to receiver. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17776) Get rid of non-binary boolean type
[
https://issues.apache.org/jira/browse/IGNITE-17776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Mashenkov updated IGNITE-17776:
--
Attachment: image.png
> Get rid of non-binary boolean type
> --
>
> Key: IGNITE-17776
> URL: https://issues.apache.org/jira/browse/IGNITE-17776
> Project: Ignite
> Issue Type: Improvement
>Reporter: Andrey Mashenkov
>Priority: Major
> Labels: ignite-3, tech-debt
> Attachments: image.png
>
>
> With all the tolerance and respect, but... for the god sake,
> let's refactor the inner class
> {{org.apache.ignite.internal.pagememory.tree.BplusTree.Bool}}
> to get rid of non-binary 4-way boolean type.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17435) Fix Datastreamer documentation.
[ https://issues.apache.org/jira/browse/IGNITE-17435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Steshin updated IGNITE-17435: -- Description: The doc should mention something like: Setting the `allowOverwrite` property to false is best used for initial data load on a cluster with stable topology. When cluster works under load, some actions may cause issues with data consistency. To avoid them: * Avoid having the same keys repeating in the data being streamed; * Avoid concurrent updates of the cache that is being streamed into; * Avoid rebalance on that cache. * Avoid snapshotting. Also, the docs should note streamer's limitations and traits. was: The doc should mention something like: Setting the allowOverwrite property to false is best used for initial data load on a cluster with stable topology. When cluster works under load, some actions may cause issues with data consistency. To avoid them: * Avoid having the same keys repeating in the data being streamed; * Avoid concurrent updates of the cache that is being streamed into; * Avoid rebalance on that cache. * Avoid snapshotting. Also, we should describe existing updaters like BatchedReceiver, BatchedSorted. > Fix Datastreamer documentation. > --- > > Key: IGNITE-17435 > URL: https://issues.apache.org/jira/browse/IGNITE-17435 > Project: Ignite > Issue Type: Sub-task >Reporter: Vladimir Steshin >Assignee: Vladimir Steshin >Priority: Major > Labels: ise > > The doc should mention something like: > Setting the `allowOverwrite` property to false is best used for initial data > load on a cluster with stable topology. When cluster works under load, some > actions may cause issues with data consistency. To avoid them: > * Avoid having the same keys repeating in the data being streamed; > * Avoid concurrent updates of the cache that is being streamed into; > * Avoid rebalance on that cache. > * Avoid snapshotting. > Also, the docs should note streamer's limitations and traits. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17435) Fix Datastreamer documentation.
[ https://issues.apache.org/jira/browse/IGNITE-17435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Steshin updated IGNITE-17435: -- Priority: Minor (was: Major) > Fix Datastreamer documentation. > --- > > Key: IGNITE-17435 > URL: https://issues.apache.org/jira/browse/IGNITE-17435 > Project: Ignite > Issue Type: Sub-task >Reporter: Vladimir Steshin >Assignee: Vladimir Steshin >Priority: Minor > Labels: ise > > The doc should mention something like: > Setting the `allowOverwrite` property to false is best used for initial data > load on a cluster with stable topology. When cluster works under load, some > actions may cause issues with data consistency. To avoid them: > * Avoid having the same keys repeating in the data being streamed; > * Avoid concurrent updates of the cache that is being streamed into; > * Avoid rebalance on that cache. > * Avoid snapshotting. > Also, the docs should note streamer's limitations and traits. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17776) Get rid of non-binary boolean type
Andrey Mashenkov created IGNITE-17776:
-
Summary: Get rid of non-binary boolean type
Key: IGNITE-17776
URL: https://issues.apache.org/jira/browse/IGNITE-17776
Project: Ignite
Issue Type: Improvement
Reporter: Andrey Mashenkov
With all the tolerance and respect, but... for the god sake,
let's refactor the inner class
{{org.apache.ignite.internal.pagememory.tree.BplusTree.Bool}}
to get rid of non-binary 4-way boolean type.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-17083) Universal full rebalance procedure for MV storage
[
https://issues.apache.org/jira/browse/IGNITE-17083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Bessonov reassigned IGNITE-17083:
--
Assignee: Ivan Bessonov
> Universal full rebalance procedure for MV storage
> -
>
> Key: IGNITE-17083
> URL: https://issues.apache.org/jira/browse/IGNITE-17083
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Assignee: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
>
> Canonical way to make "full rebalance" in RAFT is to have a persisted
> snapshots of data. This is not always a good idea. First of all, for
> persistent data is already stored somewhere and can be read at any time.
> Second, for volatile storage this requirement is just absurd.
> So, a "rebalance snapshot" should be streamed from one node to another
> instead of being written to a storage. What's good is that this approach can
> be implemented independently from the storage engine (with few adjustments to
> storage API, of course).
> h2. General idea
> Once a "rebalance snapshot" operation is triggered, we open a special type of
> cursor from the partition storage, that is able to give us all versioned
> chains in {_}some fixed order{_}. Every time the next chain has been read,
> it's remembered as the last read (let's call it\{{ lastRowId}} for now). Then
> all versions for the specific row id should be sent to receiver node in
> "Oldest to Newest" order to simplify insertion.
> This works fine without concurrent load. To account for that we need to have
> a additional collection of row ids, associated with a snapshot. Let's call it
> {{{}overwrittenRowIds{}}}.
> With this in mind, every write command should look similar to this:
> {noformat}
> for (var rebalanceSnaphot : ongoingRebalanceSnapshots) {
> try (var lock = rebalanceSnaphot.lock()) {
> if (rowId <= rebalanceSnaphot.lastRowId())
> continue;
> if (!rebalanceSnaphot.overwrittenRowIds().put(rowId))
> continue;
> rebalanceSnapshot.sendRowToReceiver(rowId);
> }
> }
> // Now modification can be freely performed.
> // Snapshot itself will skip everything from the "overwrittenRowIds"
> collection.{noformat}
> NOTE: rebalance snapshot scan must also return uncommitted write intentions.
> Their commit will be replicated later from the RAFT log.
> NOTE: receiving side will have to rebuild indexes during the rebalancing.
> Just like it works in Ignite 2.x.
> NOTE: Technically it is possible to have several nodes entering the cluster
> that require a full rebalance. So, while triggering a rebalance snapshot
> cursor, we could wait for other nodes that might want to read the same data
> and process all of them with a single scan. This is an optimization,
> obviously.
> h2. Implementation
> The implementation will have to be split into several parts, because we need:
> * Support for snapshot streaming in RAFT state machine.
> * Storage API for this type of scan.
> * Every storage must implement the new scan method.
> * Streamer itself should be implemented, along with a specific logic in
> write commands.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17742) Trying to insert a large entry in an atomic cache results in CorruptedTreeException
[
https://issues.apache.org/jira/browse/IGNITE-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610597#comment-17610597
]
Vladislav Pyatkov commented on IGNITE-17742:
LGTM
> Trying to insert a large entry in an atomic cache results in
> CorruptedTreeException
> ---
>
> Key: IGNITE-17742
> URL: https://issues.apache.org/jira/browse/IGNITE-17742
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.13
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case of using Ignite Native Persistence, the attempt to insert a new entry
> which size is greater than WAL buffer size leads to CorruptedTreeException:
> {noformat}
>
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> B+Tree is corrupted [groupId=-1407396309, pageIds=[1125904201809926],
> msg=Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=1,
> val=1, hasValBytes=true], hash=1, cacheId=0]]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.corruptedTreeException(BPlusTree.java:6487)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:2202)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1698)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1681)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2762)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:425)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1975)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2554)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2014)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1833)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1706)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:300)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:481)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:441)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:249)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1151)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:619)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2487)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2466)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1332)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:867)
> ~[classes/:?]
> at
> org.apache.ignite.internal.processors.database.IgniteDbPutGetAbstractTest.lambda$testPutLargeEntry$4bb74d57$1(IgniteDbPutGetAbstractTest.java:331)
> ~[test-classes/:?]
> at
> org.apache.ignite.internal.util.lang.RunnableX.run(RunnableX.java:37)
> ~[classes/:?]
> at
> org.apache.ignite.testframework.GridTestUtils.lambda$assertThrows$0(GridTestUtils.java:467)
>
[jira] [Assigned] (IGNITE-17772) Update documentation of spark-3.2 extension
[ https://issues.apache.org/jira/browse/IGNITE-17772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Gusev reassigned IGNITE-17772: --- Assignee: Igor Gusev > Update documentation of spark-3.2 extension > --- > > Key: IGNITE-17772 > URL: https://issues.apache.org/jira/browse/IGNITE-17772 > Project: Ignite > Issue Type: Task > Components: documentation >Reporter: Ivan Gagarkin >Assignee: Igor Gusev >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17733) Resume honest index lock
[
https://issues.apache.org/jira/browse/IGNITE-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladislav Pyatkov updated IGNITE-17733:
---
Description:
Due to an issue in protocol of lock manager, locks on indexes are avoided. Need
to find a way where we can use honest lock modes on indexes, but not convert
the type to {_}NL{_}.
was:
Due to an issue in protocol of lock manager, locks on indexes are avoided. Need
to find a way where we can use honest lock modes on indexes, but not convert
the type to {_}NAL{_}.
> Resume honest index lock
>
>
> Key: IGNITE-17733
> URL: https://issues.apache.org/jira/browse/IGNITE-17733
> Project: Ignite
> Issue Type: Bug
>Reporter: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3
>
> Due to an issue in protocol of lock manager, locks on indexes are avoided.
> Need to find a way where we can use honest lock modes on indexes, but not
> convert the type to {_}NL{_}.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17732) Only keys should be locked in transaction
[
https://issues.apache.org/jira/browse/IGNITE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladislav Pyatkov updated IGNITE-17732:
---
Description:
The transaction protocol requires locks on different entities (keys, rowIds,
tables and indexes), but our implementation ready is appliable for only one
type.
By the reason, a transaction will lock only keys (PK of those rows which
participate in the transaction). All another locks will be temporarily
mitigated.
As a temporary workaround for this disadvantage, supposed:
# Create a NL (Not a lock) lock mode, which is compatible for all another mode
# All lock modes on rowIds, tables, indexes should convert to NL
was:
The transaction protocol requires locks over different entities (keys, rowIds,
tables and indexies) all indexes (now it is a pk) in the same manner as it
happens for data (locks on {_}RowIds{_}). But lock manager cannot resolve all
problems in his case (in particular, waiter for _RowId_ does not create when
the lock on index have not got).
As a temporary workaround for this disadvantage, supposed:
# Create a NAL (Not a lock) lock mode, which is compatible for all another mode
# All lock modes on PK should convert to NAL
> Only keys should be locked in transaction
> -
>
> Key: IGNITE-17732
> URL: https://issues.apache.org/jira/browse/IGNITE-17732
> Project: Ignite
> Issue Type: Improvement
>Reporter: Vladislav Pyatkov
>Assignee: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3
>
> The transaction protocol requires locks on different entities (keys, rowIds,
> tables and indexes), but our implementation ready is appliable for only one
> type.
> By the reason, a transaction will lock only keys (PK of those rows which
> participate in the transaction). All another locks will be temporarily
> mitigated.
> As a temporary workaround for this disadvantage, supposed:
> # Create a NL (Not a lock) lock mode, which is compatible for all another
> mode
> # All lock modes on rowIds, tables, indexes should convert to NL
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17425) C++ 3.0: Basic C++ client
[ https://issues.apache.org/jira/browse/IGNITE-17425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610572#comment-17610572 ] Aleksey Demakov commented on IGNITE-17425: -- LGTM > C++ 3.0: Basic C++ client > - > > Key: IGNITE-17425 > URL: https://issues.apache.org/jira/browse/IGNITE-17425 > Project: Ignite > Issue Type: New Feature > Components: thin client >Reporter: Igor Sapego >Assignee: Igor Sapego >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > > Implement basic C++ 3.0 client that can be used as a base to add new features. > What should be done: > * Set up project structure (CMake + consider Conan); > * Establish code style; > * Document project building; > * Target C++14?; > * Set up tests (consider GTest for testing); > * Implement networking (consider adopting some library, e.g. asio); > * Implement minimal set of operations: get table by name, upsert, get; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17775) Invalid data in network buffers causes message deserialization errors and messages loss
[
https://issues.apache.org/jira/browse/IGNITE-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denis Chudov updated IGNITE-17775:
--
Description:
In some tests I observe network messages' deserialization errors and timeout
exceptions while waiting for response. In some cases there is negative group
type of the message, and this causes error:
{code:java}
java.lang.AssertionError: message type must not be negative, messageType=-5376
at
org.apache.ignite.network.MessageSerializationRegistryImpl.getFactory(MessageSerializationRegistryImpl.java:77)
at
org.apache.ignite.network.MessageSerializationRegistryImpl.createDeserializer(MessageSerializationRegistryImpl.java:102)
at
org.apache.ignite.internal.network.serialization.SerializationService.createDeserializer(SerializationService.java:68)
at
org.apache.ignite.internal.network.serialization.PerSessionSerializationService.createMessageDeserializer(PerSessionSerializationService.java:109)
at
org.apache.ignite.internal.network.netty.InboundDecoder.decode(InboundDecoder.java:89)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
{code}
When the group or message type is positive but not existing, there should be a
NetworkConfigurationException but it's not displayed in logs, however, it
causes TimeoutExceptions because of messages loss.
This reproduces in
https://github.com/gridgain/apache-ignite-3/tree/ignite-17523-2 in
ItTablesApiTest#testGetTableFromLaggedNode
was:
In some tests I observe network messages' deserialization errors and timeout
exceptions while waiting for response. In some cases there is negative group
type of the message, and this causes error:
{code:java}
java.lang.AssertionError: message type must not be negative, messageType=-5376
at
org.apache.ignite.network.MessageSerializationRegistryImpl.getFactory(MessageSerializationRegistryImpl.java:77)
at
org.apache.ignite.network.MessageSerializationRegistryImpl.createDeserializer(MessageSerializationRegistryImpl.java:102)
at
org.apache.ignite.internal.network.serialization.SerializationService.createDeserializer(SerializationService.java:68)
at
org.apache.ignite.internal.network.serialization.PerSessionSerializationService.createMessageDeserializer(PerSessionSerializationService.java:109)
at
org.apache.ignite.internal.network.netty.InboundDecoder.decode(InboundDecoder.java:89)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
[jira] [Updated] (IGNITE-17775) Invalid data in network buffers causes message deserialization errors and messages loss
[
https://issues.apache.org/jira/browse/IGNITE-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denis Chudov updated IGNITE-17775:
--
Issue Type: Bug (was: Task)
> Invalid data in network buffers causes message deserialization errors and
> messages loss
> ---
>
> Key: IGNITE-17775
> URL: https://issues.apache.org/jira/browse/IGNITE-17775
> Project: Ignite
> Issue Type: Bug
>Reporter: Denis Chudov
>Priority: Major
> Labels: ignite-3
>
> In some tests I observe network messages' deserialization errors and timeout
> exceptions while waiting for response. In some cases there is negative group
> type of the message, and this causes error:
> {code:java}
> java.lang.AssertionError: message type must not be negative, messageType=-5376
> at
> org.apache.ignite.network.MessageSerializationRegistryImpl.getFactory(MessageSerializationRegistryImpl.java:77)
> at
> org.apache.ignite.network.MessageSerializationRegistryImpl.createDeserializer(MessageSerializationRegistryImpl.java:102)
> at
> org.apache.ignite.internal.network.serialization.SerializationService.createDeserializer(SerializationService.java:68)
> at
> org.apache.ignite.internal.network.serialization.PerSessionSerializationService.createMessageDeserializer(PerSessionSerializationService.java:109)
> at
> org.apache.ignite.internal.network.netty.InboundDecoder.decode(InboundDecoder.java:89)
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:833)
> {code}
> When the group or message type is positive but not existing, there should be
> a NetworkConfigurationException but it's not displayed in logs, however, it
> causes TimeoutExceptions because of messages loss.
> This reproduces in
> https://github.com/gridgain/apache-ignite-3/tree/ignite-17523-2 in
> ItTablesApiTest#testGetTableFromLaggedNode
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-17775) Invalid data in network buffers causes message deserialization errors and messages loss
Denis Chudov created IGNITE-17775:
-
Summary: Invalid data in network buffers causes message
deserialization errors and messages loss
Key: IGNITE-17775
URL: https://issues.apache.org/jira/browse/IGNITE-17775
Project: Ignite
Issue Type: Task
Reporter: Denis Chudov
In some tests I observe network messages' deserialization errors and timeout
exceptions while waiting for response. In some cases there is negative group
type of the message, and this causes error:
{code:java}
java.lang.AssertionError: message type must not be negative, messageType=-5376
at
org.apache.ignite.network.MessageSerializationRegistryImpl.getFactory(MessageSerializationRegistryImpl.java:77)
at
org.apache.ignite.network.MessageSerializationRegistryImpl.createDeserializer(MessageSerializationRegistryImpl.java:102)
at
org.apache.ignite.internal.network.serialization.SerializationService.createDeserializer(SerializationService.java:68)
at
org.apache.ignite.internal.network.serialization.PerSessionSerializationService.createMessageDeserializer(PerSessionSerializationService.java:109)
at
org.apache.ignite.internal.network.netty.InboundDecoder.decode(InboundDecoder.java:89)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
{code}
When the group or message type is positive but not existing, there should be a
NetworkConfigurationException but it's not displayed in logs, but causes
TimeoutExceptions because of messages loss.
This reproduces in
https://github.com/gridgain/apache-ignite-3/tree/ignite-17523-2 in
ItTablesApiTest#testGetTableFromLaggedNode
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17132) [Native Persistence 3.0] Implement partition destruction for persistent PageMemory
[ https://issues.apache.org/jira/browse/IGNITE-17132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17132: - Epic Link: IGNITE-17774 (was: IGNITE-16232) > [Native Persistence 3.0] Implement partition destruction for persistent > PageMemory > -- > > Key: IGNITE-17132 > URL: https://issues.apache.org/jira/browse/IGNITE-17132 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > > At the moment, the partition destruction operation does not work quite > correctly, we just go through all the records and explicitly delete them, but > we cannot delete the partition file, because the checkpoint will then not be > able to write pages to this partition file and it will have an error. > Before implementation, you need to think about how to do it, but in any case, > deleting the partition file should definitely do a checkpoint. > See: > *org.apache.ignite.internal.storage.pagememory.PersistentPageMemoryPartitionStorage#destroy* > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17132) [Native Persistence 3.0] Implement partition destruction for persistent PageMemory
[ https://issues.apache.org/jira/browse/IGNITE-17132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17132: - Fix Version/s: (was: 3.0.0-alpha6) > [Native Persistence 3.0] Implement partition destruction for persistent > PageMemory > -- > > Key: IGNITE-17132 > URL: https://issues.apache.org/jira/browse/IGNITE-17132 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > > At the moment, the partition destruction operation does not work quite > correctly, we just go through all the records and explicitly delete them, but > we cannot delete the partition file, because the checkpoint will then not be > able to write pages to this partition file and it will have an error. > Before implementation, you need to think about how to do it, but in any case, > deleting the partition file should definitely do a checkpoint. > See: > *org.apache.ignite.internal.storage.pagememory.PersistentPageMemoryPartitionStorage#destroy* > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-17773) Remove node start/stop/list commands
[ https://issues.apache.org/jira/browse/IGNITE-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksandr reassigned IGNITE-17773: -- Assignee: Aleksandr > Remove node start/stop/list commands > > > Key: IGNITE-17773 > URL: https://issues.apache.org/jira/browse/IGNITE-17773 > Project: Ignite > Issue Type: Task > Components: cli >Reporter: Aleksandr >Assignee: Aleksandr >Priority: Major > Labels: ignite-3 > > Now, ignite3 node start/stop/restart managements are done through a special > script that is provided by the distribution. CLI commands: node start, node > stop, node list, and bootstrap now do not make any sense. We have to remove > them because they do not work if a user has started a node with a startup > script. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17262) Implement RAFT snapshot streamer
[ https://issues.apache.org/jira/browse/IGNITE-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17262: - Epic Link: IGNITE-17774 (was: IGNITE-16923) > Implement RAFT snapshot streamer > > > Key: IGNITE-17262 > URL: https://issues.apache.org/jira/browse/IGNITE-17262 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > Infrastructure for this should be implemented in IGNITE-17253 . > In this ticket, streaming using this infrastructure (along with changes to > write commands/their handling) should be implemented. > See IGNITE-17083 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17262) Implement RAFT snapshot streamer
[ https://issues.apache.org/jira/browse/IGNITE-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17262: - Fix Version/s: (was: 3.0.0-alpha6) > Implement RAFT snapshot streamer > > > Key: IGNITE-17262 > URL: https://issues.apache.org/jira/browse/IGNITE-17262 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > Infrastructure for this should be implemented in IGNITE-17253 . > In this ticket, streaming using this infrastructure (along with changes to > write commands/their handling) should be implemented. > See IGNITE-17083 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17264) Support RAFT snapshot streaming for RocksDB storage
[ https://issues.apache.org/jira/browse/IGNITE-17264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17264: - Epic Link: IGNITE-17774 (was: IGNITE-16923) > Support RAFT snapshot streaming for RocksDB storage > --- > > Key: IGNITE-17264 > URL: https://issues.apache.org/jira/browse/IGNITE-17264 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > IGNITE-17254 API needs to be implemented for RocksDB storage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17264) Support RAFT snapshot streaming for RocksDB storage
[ https://issues.apache.org/jira/browse/IGNITE-17264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17264: - Fix Version/s: (was: 3.0.0-alpha6) > Support RAFT snapshot streaming for RocksDB storage > --- > > Key: IGNITE-17264 > URL: https://issues.apache.org/jira/browse/IGNITE-17264 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > IGNITE-17254 API needs to be implemented for RocksDB storage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17253) Support for snapshot streaming in RAFT state machine
[ https://issues.apache.org/jira/browse/IGNITE-17253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17253: - Fix Version/s: (was: 3.0.0-alpha6) > Support for snapshot streaming in RAFT state machine > > > Key: IGNITE-17253 > URL: https://issues.apache.org/jira/browse/IGNITE-17253 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > In JRaft, snapshotting works through files: first, a snapshot is written to a > file, later it is sent to follower(s). > This requires additional FS space, and it is a bit strange for in-memory > cases. > We need to switch to streaming way of doing snapshots: when the leader needs > to bootstrap a follower, it opens a 'channel' to the follower, starts > producing a snapshot and streaming it to the follower. The follower applies > the snapshot in the same streaming way. > This ticket is about modifying JRaft internal infrastructure to switch to > snapshot streaming. > See IGNITE-17083 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17253) Support for snapshot streaming in RAFT state machine
[ https://issues.apache.org/jira/browse/IGNITE-17253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17253: - Epic Link: IGNITE-17774 (was: IGNITE-16923) > Support for snapshot streaming in RAFT state machine > > > Key: IGNITE-17253 > URL: https://issues.apache.org/jira/browse/IGNITE-17253 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > > In JRaft, snapshotting works through files: first, a snapshot is written to a > file, later it is sent to follower(s). > This requires additional FS space, and it is a bit strange for in-memory > cases. > We need to switch to streaming way of doing snapshots: when the leader needs > to bootstrap a follower, it opens a 'channel' to the follower, starts > producing a snapshot and streaming it to the follower. The follower applies > the snapshot in the same streaming way. > This ticket is about modifying JRaft internal infrastructure to switch to > snapshot streaming. > See IGNITE-17083 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17254) Storage API for RAFT snapshot streaming
[ https://issues.apache.org/jira/browse/IGNITE-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17254: - Fix Version/s: (was: 3.0.0-alpha6) > Storage API for RAFT snapshot streaming > --- > > Key: IGNITE-17254 > URL: https://issues.apache.org/jira/browse/IGNITE-17254 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > Current PartitionStorage has a method for taking snapshots (and a method for > restoring from them). The methods work with files. In IGNITE-17253, we are > going to switch to streaming protocol (which does not use files at all). So > we need streaming snapshot-related methods on MvPartitionStorage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17254) Storage API for RAFT snapshot streaming
[ https://issues.apache.org/jira/browse/IGNITE-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17254: - Epic Link: IGNITE-17774 (was: IGNITE-16923) > Storage API for RAFT snapshot streaming > --- > > Key: IGNITE-17254 > URL: https://issues.apache.org/jira/browse/IGNITE-17254 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > > Current PartitionStorage has a method for taking snapshots (and a method for > restoring from them). The methods work with files. In IGNITE-17253, we are > going to switch to streaming protocol (which does not use files at all). So > we need streaming snapshot-related methods on MvPartitionStorage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17084) Native rebalance for RocksDB partitions
[ https://issues.apache.org/jira/browse/IGNITE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17084: - Epic Link: IGNITE-17774 (was: IGNITE-16923) > Native rebalance for RocksDB partitions > --- > > Key: IGNITE-17084 > URL: https://issues.apache.org/jira/browse/IGNITE-17084 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Bessonov >Priority: Major > Labels: ignite-3 > > General idea of full rebalance is described in > https://issues.apache.org/jira/browse/IGNITE-17083 > For persistent storages, there's an option to avoid copy-on-write rebalance > algorithms if it's desired. Intuitively, it's a preferable option. Each > storage chooses its own format. > In this case, RocksDB allows consistent db iteration using a "Snapshot" > feature. Idea is very simple: > * Take a RoackDB snapshot. > * Iterate through partition data. > * Iterate through indexes. > * Relese the snapshot. > There must be a common "infrastructure" or a framework to stream native > rebalance snapshots. Data format should be as simple as possible. > NOTE: of course, it has to be mentioned that this approach might lead to > ineffective storage space usage. What I mean is that "previous" versions of > values, in terms of RocksDB, must be stored on the device if they're visible > from any of snapshots. It can be a problem in theory, but in practice full > rebalance isn't expected to occur often, and event then we don't expect that > users will rewrite the entire partition data in a span of a single rebalance. > h2. Possible problems > Given that "raw" data is sent, including sql indexes, all incompleted indexes > will be sent incompleted. Maybe we should also send a build state for each > index so that the receiving side could continue from the right place, not > from the beginning. > This problem will be resolved in the future. Currently we don't have indexes > implemented. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17364) Move control utility index tasks to the core module
[ https://issues.apache.org/jira/browse/IGNITE-17364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pavlov updated IGNITE-17364: --- Labels: ignite-osgi ise (was: ignite-osgi) > Move control utility index tasks to the core module > --- > > Key: IGNITE-17364 > URL: https://issues.apache.org/jira/browse/IGNITE-17364 > Project: Ignite > Issue Type: Improvement >Reporter: Aleksey Plekhanov >Assignee: Aleksey Plekhanov >Priority: Major > Labels: ignite-osgi, ise > Fix For: 2.14 > > Time Spent: 0.5h > Remaining Estimate: 0h > > There are some control utility tasks (indexes list, index validate, index > rebuild), which are depended on H2 and located in indexing module. But > instead of H2 module classes core module classes having almost the same > functionality can be used. > We should rewrite these tasks to use core module classes to make tasks > workable without H2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17087) Native rebalance for PDS partitions
[
https://issues.apache.org/jira/browse/IGNITE-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Chugunov updated IGNITE-17087:
-
Epic Link: IGNITE-17774 (was: IGNITE-16923)
> Native rebalance for PDS partitions
> ---
>
> Key: IGNITE-17087
> URL: https://issues.apache.org/jira/browse/IGNITE-17087
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
>
> General idea of full rebalance is described in
> https://issues.apache.org/jira/browse/IGNITE-17083
> For persistent storages, there's an option to avoid copy-on-write rebalance
> algorithms if it's desired. Intuitively, it's a preferable option. Each
> storage chooses its own format.
> h2. General idea
> In this case, PDS has checkpointing feature that saves consistent state on
> disk. I expect SQL indexes to be in the same partition file as other data.
> For every partition, its state on disk would look like this:
> {code:java}
> part-x.bin
> part-x-1.bin
> part-x-2.bin
> ...
> part-x-n.bin{code}
> part-x.bin is a baseline, and every other file is a delta that should be
> applied to underlying layers to get consistent data. It can be viewed like
> full and incremental backups.
> When rebalance snapshot is required, we could force a checkpoint and then
> *prohibit merging* of new deltas to delta files from the snapshot until
> rebalance is finished. We must guarantee that consistent state can be read
> from disk.
> Now, there are several strategies of data transferring:
> * File-based. We can send baseline and delta files as files. Two possible
> issues here:
> ** Files contain duplicated pages, so the volume of data will be bigger than
> necessary.
> ** Baseline file has to be truncated, because some delta pages go directly
> into baseline file as optimization.
> * Page-based. Latest state of every required page is sent separately. Two
> strategies here:
> ** Iterate pages in order of page indexes. Overheads during reads, but
> writes are very effective.
> ** Iterate pages in order of delta files, skipping already read pages in the
> process (like snapshots in GridGain, for example). Little overhead on read,
> but write won't be append-only.
> I would argue that slower reads are more appropriate then slower writes.
> Generally speaking, any write should be slower than any read of the same
> size, right?
> Should we implement all strategies and give user a choice? It's hard to
> predict which one is better for which scenario. In the future, I think it
> would be convenient to implement many options, but at first we should stick
> to the simplest one.
> There must be a common "infrastructure" or a framework to stream native
> rebalance snapshots. Data format should be as simple as possible.
> NOTE: of course, it has to be mentioned that this approach might lead to
> ineffective storage space usage. It can be a problem in theory, but in
> practice full rebalance isn't expected to occur often, and event then we
> don't expect that users will rewrite the entire partition data in a span of a
> single rebalance.
> h2. Possible problems
> Given that "raw" data is sent, including sql indexes, all incompleted indexes
> will be sent incompleted. Maybe we should also send a build state for each
> index so that the receiving side could continue from the right place, not
> from the beginning.
> This problem will be resolved in the future. Currently we don't have indexes
> implemented.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17083) Universal full rebalance procedure for MV storage
[
https://issues.apache.org/jira/browse/IGNITE-17083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Chugunov updated IGNITE-17083:
-
Epic Link: IGNITE-17774 (was: IGNITE-16923)
> Universal full rebalance procedure for MV storage
> -
>
> Key: IGNITE-17083
> URL: https://issues.apache.org/jira/browse/IGNITE-17083
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
>
> Canonical way to make "full rebalance" in RAFT is to have a persisted
> snapshots of data. This is not always a good idea. First of all, for
> persistent data is already stored somewhere and can be read at any time.
> Second, for volatile storage this requirement is just absurd.
> So, a "rebalance snapshot" should be streamed from one node to another
> instead of being written to a storage. What's good is that this approach can
> be implemented independently from the storage engine (with few adjustments to
> storage API, of course).
> h2. General idea
> Once a "rebalance snapshot" operation is triggered, we open a special type of
> cursor from the partition storage, that is able to give us all versioned
> chains in {_}some fixed order{_}. Every time the next chain has been read,
> it's remembered as the last read (let's call it\{{ lastRowId}} for now). Then
> all versions for the specific row id should be sent to receiver node in
> "Oldest to Newest" order to simplify insertion.
> This works fine without concurrent load. To account for that we need to have
> a additional collection of row ids, associated with a snapshot. Let's call it
> {{{}overwrittenRowIds{}}}.
> With this in mind, every write command should look similar to this:
> {noformat}
> for (var rebalanceSnaphot : ongoingRebalanceSnapshots) {
> try (var lock = rebalanceSnaphot.lock()) {
> if (rowId <= rebalanceSnaphot.lastRowId())
> continue;
> if (!rebalanceSnaphot.overwrittenRowIds().put(rowId))
> continue;
> rebalanceSnapshot.sendRowToReceiver(rowId);
> }
> }
> // Now modification can be freely performed.
> // Snapshot itself will skip everything from the "overwrittenRowIds"
> collection.{noformat}
> NOTE: rebalance snapshot scan must also return uncommitted write intentions.
> Their commit will be replicated later from the RAFT log.
> NOTE: receiving side will have to rebuild indexes during the rebalancing.
> Just like it works in Ignite 2.x.
> NOTE: Technically it is possible to have several nodes entering the cluster
> that require a full rebalance. So, while triggering a rebalance snapshot
> cursor, we could wait for other nodes that might want to read the same data
> and process all of them with a single scan. This is an optimization,
> obviously.
> h2. Implementation
> The implementation will have to be split into several parts, because we need:
> * Support for snapshot streaming in RAFT state machine.
> * Storage API for this type of scan.
> * Every storage must implement the new scan method.
> * Streamer itself should be implemented, along with a specific logic in
> write commands.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17774) Storage to provide API and features to support data rebalancing
[ https://issues.apache.org/jira/browse/IGNITE-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17774: - Ignite Flags: (was: Docs Required,Release Notes Required) > Storage to provide API and features to support data rebalancing > --- > > Key: IGNITE-17774 > URL: https://issues.apache.org/jira/browse/IGNITE-17774 > Project: Ignite > Issue Type: Epic > Components: persistence >Reporter: Sergey Chugunov >Priority: Major > > Storage component should provide APIs and implementations for all types of > storages (Page Memory, Rocks DB) to support rebalancing of data between nodes. > This includes several features: > # Rebalancing preparation - consistent snapshots of stored data. > # Effective streaming of snapshots - both on sender and receiver sides. > # Cleaning up resources when rebalancing is completed. > Integration with indexes and SQL engine is out of scope. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17773) Remove node start/stop/list commands
Aleksandr created IGNITE-17773: -- Summary: Remove node start/stop/list commands Key: IGNITE-17773 URL: https://issues.apache.org/jira/browse/IGNITE-17773 Project: Ignite Issue Type: Task Components: cli Reporter: Aleksandr Now, ignite3 node start/stop/restart managements are done through a special script that is provided by the distribution. CLI commands: node start, node stop, node list, and bootstrap now do not make any sense. We have to remove them because they do not work if a user has started a node with a startup script. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17774) Storage to provide API and features to support data rebalancing
Sergey Chugunov created IGNITE-17774: Summary: Storage to provide API and features to support data rebalancing Key: IGNITE-17774 URL: https://issues.apache.org/jira/browse/IGNITE-17774 Project: Ignite Issue Type: Epic Components: persistence Reporter: Sergey Chugunov Storage component should provide APIs and implementations for all types of storages (Page Memory, Rocks DB) to support rebalancing of data between nodes. This includes several features: # Rebalancing preparation - consistent snapshots of stored data. # Effective streaming of snapshots - both on sender and receiver sides. # Cleaning up resources when rebalancing is completed. Integration with indexes and SQL engine is out of scope. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17755) Add common interface for sorted and hash indices
[
https://issues.apache.org/jira/browse/IGNITE-17755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Polovtcev updated IGNITE-17755:
-
Reviewer: Semyon Danilov
> Add common interface for sorted and hash indices
>
>
> Key: IGNITE-17755
> URL: https://issues.apache.org/jira/browse/IGNITE-17755
> Project: Ignite
> Issue Type: Task
>Reporter: Aleksandr Polovtcev
>Assignee: Aleksandr Polovtcev
>Priority: Major
> Labels: ignite-3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Create an interface with some common methods across all index implementations
> so that it would be possible to work with indices without knowing the actual
> index type.
> A new interface may look like this:
> {code:java}
> public interface IndexStorage {
> Cursor get(BinaryTuple key) throws StorageException;
> void put(IndexRow row) throws StorageException;
> void remove(IndexRow row) throws StorageException;
> }
> {code}
> Also, a new method should be added to {{MvTableStorage}}:
> {code:java}
> IndexStorage getIndex(int partitionId, UUID indexId);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17711) Change Binary Tuple Prefix format to allow comparison without deserialization
[ https://issues.apache.org/jira/browse/IGNITE-17711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610511#comment-17610511 ] Roman Puchkovskiy commented on IGNITE-17711: The patch looks good to me > Change Binary Tuple Prefix format to allow comparison without deserialization > - > > Key: IGNITE-17711 > URL: https://issues.apache.org/jira/browse/IGNITE-17711 > Project: Ignite > Issue Type: Improvement >Reporter: Aleksandr Polovtcev >Assignee: Aleksandr Polovtcev >Priority: Minor > Labels: ignite-3 > Time Spent: 0.5h > Remaining Estimate: 0h > > BinaryTuplePrefix format dictates that the number of prefix elements are > appended to the end of the tuple. This is currently implemented as inserting > an extra int element which also implies an extra entry in the offset map and > non-deterministic size of this element (it can occupy from 1 to 4 bytes). > This was done for the sake of simplicity but it also has an implication: it > is not possible to simply compare it with a BinaryTuple byte-by-byte in order > to determine if a given prefix matches a given tuple. A better approach would > be to append the number of elements directly as the last two bytes (or four, > if necessary) of the tuple, bypassing the entry table. This way we can > discard the last two bytes and compare the prefix and the tuple on a per-byte > basis. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-17717) Logging cdc in ignite2ignite mode
[ https://issues.apache.org/jira/browse/IGNITE-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Izhikov reassigned IGNITE-17717: Assignee: Nikolay Izhikov > Logging cdc in ignite2ignite mode > - > > Key: IGNITE-17717 > URL: https://issues.apache.org/jira/browse/IGNITE-17717 > Project: Ignite > Issue Type: Task >Reporter: Luchnikov Alexander >Assignee: Nikolay Izhikov >Priority: Major > Labels: ise > > When using cdc in ignite2ignite mode, there is a problem with logging. > When running ignite-cdc.sh, the log is written to ignite-cdc.log until the > ignite client starts, after it starts, the recording continues to ignite.log. > Probably the problem is related to the replacement of appId at the start of > the client node. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17772) Update documentation of spark-3.2 extension
[ https://issues.apache.org/jira/browse/IGNITE-17772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Gagarkin updated IGNITE-17772: --- Summary: Update documentation of spark-3.2 extension (was: Update documentation of spark-3.2) > Update documentation of spark-3.2 extension > --- > > Key: IGNITE-17772 > URL: https://issues.apache.org/jira/browse/IGNITE-17772 > Project: Ignite > Issue Type: Task > Components: documentation >Reporter: Ivan Gagarkin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17772) Update documentation of spark-3.2
Ivan Gagarkin created IGNITE-17772: -- Summary: Update documentation of spark-3.2 Key: IGNITE-17772 URL: https://issues.apache.org/jira/browse/IGNITE-17772 Project: Ignite Issue Type: Task Components: documentation Reporter: Ivan Gagarkin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17495) Create start, stop, restart scripts for ingite-db
[ https://issues.apache.org/jira/browse/IGNITE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-17495: Fix Version/s: 3.0.0-alpha6 > Create start, stop, restart scripts for ingite-db > - > > Key: IGNITE-17495 > URL: https://issues.apache.org/jira/browse/IGNITE-17495 > Project: Ignite > Issue Type: Improvement > Components: build >Reporter: Mikhail Pochatkin >Assignee: Aleksandr >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17488) Packaging: Zip archive
[ https://issues.apache.org/jira/browse/IGNITE-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-17488: Fix Version/s: 3.0.0-alpha6 > Packaging: Zip archive > --- > > Key: IGNITE-17488 > URL: https://issues.apache.org/jira/browse/IGNITE-17488 > Project: Ignite > Issue Type: New Feature > Components: build >Reporter: Mikhail Pochatkin >Assignee: Aleksandr >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > > Create zip archive target with all Apache Ignite binaries and > install\uninstall\update scripts. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-17488) Packaging: Zip archive
[ https://issues.apache.org/jira/browse/IGNITE-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov resolved IGNITE-17488. - Resolution: Fixed > Packaging: Zip archive > --- > > Key: IGNITE-17488 > URL: https://issues.apache.org/jira/browse/IGNITE-17488 > Project: Ignite > Issue Type: New Feature > Components: build >Reporter: Mikhail Pochatkin >Assignee: Aleksandr >Priority: Major > Labels: ignite-3 > > Create zip archive target with all Apache Ignite binaries and > install\uninstall\update scripts. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-17495) Create start, stop, restart scripts for ingite-db
[ https://issues.apache.org/jira/browse/IGNITE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov resolved IGNITE-17495. - Resolution: Fixed The patch looks good to me. Thank you for the contribution, merged to the main branch. > Create start, stop, restart scripts for ingite-db > - > > Key: IGNITE-17495 > URL: https://issues.apache.org/jira/browse/IGNITE-17495 > Project: Ignite > Issue Type: Improvement > Components: build >Reporter: Mikhail Pochatkin >Assignee: Aleksandr >Priority: Major > Labels: ignite-3 > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17495) Create start, stop, restart scripts for ingite-db
[ https://issues.apache.org/jira/browse/IGNITE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-17495: Reviewer: Semyon Danilov > Create start, stop, restart scripts for ingite-db > - > > Key: IGNITE-17495 > URL: https://issues.apache.org/jira/browse/IGNITE-17495 > Project: Ignite > Issue Type: Improvement > Components: build >Reporter: Mikhail Pochatkin >Assignee: Aleksandr >Priority: Major > Labels: ignite-3 > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17671) Implement BinaryTuple inlining in a sorted index B+Tree
[ https://issues.apache.org/jira/browse/IGNITE-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17671: - Epic Link: IGNITE-17767 (was: IGNITE-17304) > Implement BinaryTuple inlining in a sorted index B+Tree > --- > > Key: IGNITE-17671 > URL: https://issues.apache.org/jira/browse/IGNITE-17671 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > > In a simple implementation, instead of a *BinaryTuple*, we store its link > from the *FreeList* in the key, this is not optimal and we need to inline the > *BinaryTuple* in the key, for starters, you can see how to do this in 2.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17711) Change Binary Tuple Prefix format to allow comparison without deserialization
[ https://issues.apache.org/jira/browse/IGNITE-17711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17711: - Epic Link: IGNITE-17767 (was: IGNITE-17304) > Change Binary Tuple Prefix format to allow comparison without deserialization > - > > Key: IGNITE-17711 > URL: https://issues.apache.org/jira/browse/IGNITE-17711 > Project: Ignite > Issue Type: Improvement >Reporter: Aleksandr Polovtcev >Assignee: Aleksandr Polovtcev >Priority: Minor > Labels: ignite-3 > Time Spent: 20m > Remaining Estimate: 0h > > BinaryTuplePrefix format dictates that the number of prefix elements are > appended to the end of the tuple. This is currently implemented as inserting > an extra int element which also implies an extra entry in the offset map and > non-deterministic size of this element (it can occupy from 1 to 4 bytes). > This was done for the sake of simplicity but it also has an implication: it > is not possible to simply compare it with a BinaryTuple byte-by-byte in order > to determine if a given prefix matches a given tuple. A better approach would > be to append the number of elements directly as the last two bytes (or four, > if necessary) of the tuple, bypassing the entry table. This way we can > discard the last two bytes and compare the prefix and the tuple on a per-byte > basis. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17536) Implement BinaryTuple inlining in a hash index B+Tree
[ https://issues.apache.org/jira/browse/IGNITE-17536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17536: - Epic Link: IGNITE-17767 (was: IGNITE-17304) > Implement BinaryTuple inlining in a hash index B+Tree > - > > Key: IGNITE-17536 > URL: https://issues.apache.org/jira/browse/IGNITE-17536 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > > In a simple implementation, instead of a *BinaryTuple*, we store its link > from the *FreeList* in the key, this is not optimal and we need to inline the > *BinaryTuple* in the key, for starters, you can see how to do this in 2.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17325) Consider in-place compare for BinaryTuple comparator
[ https://issues.apache.org/jira/browse/IGNITE-17325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-17325: - Epic Link: IGNITE-17767 (was: IGNITE-17304) > Consider in-place compare for BinaryTuple comparator > > > Key: IGNITE-17325 > URL: https://issues.apache.org/jira/browse/IGNITE-17325 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Bessonov >Priority: Major > Labels: ignite-3 > > We should be able to compare columns in IGNITE-17318 / IGNITE-17320 without > deserializing them. This includes String, BigInteger, BigDecimal, BitMap and > maybe other types that I forgot about -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-17027) Incorrect result of the DML delete operation, in some environment
[ https://issues.apache.org/jira/browse/IGNITE-17027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Plekhanov reassigned IGNITE-17027: -- Assignee: Aleksey Plekhanov > Incorrect result of the DML delete operation, in some environment > -- > > Key: IGNITE-17027 > URL: https://issues.apache.org/jira/browse/IGNITE-17027 > Project: Ignite > Issue Type: Bug > Components: sql >Reporter: Luchnikov Alexander >Assignee: Aleksey Plekhanov >Priority: Minor > Labels: ise > Attachments: IndexSetArgsAndCastTest.patch > > > Description of the case, in some environment, the DML operation does not > delete all data (this DML delete operation is an example). To reproduce the > problem, you must: > # Create a table with a varchar field. > # Create an index on the given field. > # Fill the table with data, use int as the value. > # Delete data by specifying in the DML operation, as a condition, an indexed > value, without String.valueOf(intValue) > The reproducer( [^IndexSetArgsAndCastTest.patch] ) shows this behavior in > indexOnAutocastOff() test. > The result of all tests should be the same, specifically in this example (DML > delete) - the number of entries in the cache, according to the result of each > test, should be equal to zero. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17717) Logging cdc in ignite2ignite mode
[ https://issues.apache.org/jira/browse/IGNITE-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pavlov updated IGNITE-17717: --- Priority: Major (was: Minor) > Logging cdc in ignite2ignite mode > - > > Key: IGNITE-17717 > URL: https://issues.apache.org/jira/browse/IGNITE-17717 > Project: Ignite > Issue Type: Task >Reporter: Luchnikov Alexander >Priority: Major > Labels: ise > > When using cdc in ignite2ignite mode, there is a problem with logging. > When running ignite-cdc.sh, the log is written to ignite-cdc.log until the > ignite client starts, after it starts, the recording continues to ignite.log. > Probably the problem is related to the replacement of appId at the start of > the client node. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-16510) Calcite engine. Support "keep binary" flag
[ https://issues.apache.org/jira/browse/IGNITE-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Plekhanov reassigned IGNITE-16510: -- Assignee: Aleksey Plekhanov > Calcite engine. Support "keep binary" flag > -- > > Key: IGNITE-16510 > URL: https://issues.apache.org/jira/browse/IGNITE-16510 > Project: Ignite > Issue Type: New Feature >Reporter: Aleksey Plekhanov >Assignee: Aleksey Plekhanov >Priority: Major > Labels: calcite, calcite2-required, calcite3-required > > The "keep binary" flag is currently ignored by the Calcite-based SQL engine > and there is no way to return a deserialized object: all POJO returned in > binary format. We should support the "keep binary" flag and deserialize > objects if needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17326) Update Ignite dependency: Spring
[ https://issues.apache.org/jira/browse/IGNITE-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pavlov updated IGNITE-17326: --- Labels: ise (was: ) > Update Ignite dependency: Spring > > > Key: IGNITE-17326 > URL: https://issues.apache.org/jira/browse/IGNITE-17326 > Project: Ignite > Issue Type: Improvement >Reporter: Aleksandr >Assignee: Aleksandr >Priority: Major > Labels: ise > Fix For: 2.14 > > Time Spent: 20m > Remaining Estimate: 0h > > Update Ignite dependency Spring 5.2.21.RELEASE to 5.2.22.RELEASE -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17323) Update Ignite dependency: Jetty
[ https://issues.apache.org/jira/browse/IGNITE-17323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pavlov updated IGNITE-17323: --- Labels: ise (was: ) > Update Ignite dependency: Jetty > --- > > Key: IGNITE-17323 > URL: https://issues.apache.org/jira/browse/IGNITE-17323 > Project: Ignite > Issue Type: Improvement >Reporter: Aleksandr >Assignee: Aleksandr >Priority: Major > Labels: ise > Fix For: 2.14 > > Time Spent: 20m > Remaining Estimate: 0h > > Update Jetty dependency 9.4.39.v20210325 to 9.4.43.v20210629 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17732) Only keys should be locked in transaction
[
https://issues.apache.org/jira/browse/IGNITE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladislav Pyatkov updated IGNITE-17732:
---
Description:
The transaction protocol requires locks over different entities (keys, rowIds,
tables and indexies) all indexes (now it is a pk) in the same manner as it
happens for data (locks on {_}RowIds{_}). But lock manager cannot resolve all
problems in his case (in particular, waiter for _RowId_ does not create when
the lock on index have not got).
As a temporary workaround for this disadvantage, supposed:
# Create a NAL (Not a lock) lock mode, which is compatible for all another mode
# All lock modes on PK should convert to NAL
was:
The transaction protocol requires lock all indexes (now it is a pk) in the same
manner as it happens for data (locks on {_}RowIds{_}). But lock manager cannot
resolve all problems in his case (in particular, waiter for _RowId_ does not
create when the lock on index have not got).
As a temporary workaround for this disadvantage, supposed:
# Create a NAL (Not a lock) lock mode, which is compatible for all another mode
# All lock modes on PK should convert to NAL
> Only keys should be locked in transaction
> -
>
> Key: IGNITE-17732
> URL: https://issues.apache.org/jira/browse/IGNITE-17732
> Project: Ignite
> Issue Type: Improvement
>Reporter: Vladislav Pyatkov
>Assignee: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3
>
> The transaction protocol requires locks over different entities (keys,
> rowIds, tables and indexies) all indexes (now it is a pk) in the same manner
> as it happens for data (locks on {_}RowIds{_}). But lock manager cannot
> resolve all problems in his case (in particular, waiter for _RowId_ does not
> create when the lock on index have not got).
> As a temporary workaround for this disadvantage, supposed:
> # Create a NAL (Not a lock) lock mode, which is compatible for all another
> mode
> # All lock modes on PK should convert to NAL
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17732) Only keys should be locked in transaction
[
https://issues.apache.org/jira/browse/IGNITE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladislav Pyatkov updated IGNITE-17732:
---
Summary: Only keys should be locked in transaction (was: Avoid locking
indexes in transaction)
> Only keys should be locked in transaction
> -
>
> Key: IGNITE-17732
> URL: https://issues.apache.org/jira/browse/IGNITE-17732
> Project: Ignite
> Issue Type: Improvement
>Reporter: Vladislav Pyatkov
>Assignee: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3
>
> The transaction protocol requires lock all indexes (now it is a pk) in the
> same manner as it happens for data (locks on {_}RowIds{_}). But lock manager
> cannot resolve all problems in his case (in particular, waiter for _RowId_
> does not create when the lock on index have not got).
> As a temporary workaround for this disadvantage, supposed:
> # Create a NAL (Not a lock) lock mode, which is compatible for all another
> mode
> # All lock modes on PK should convert to NAL
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17732) Avoid locking indexes in transaction
[
https://issues.apache.org/jira/browse/IGNITE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610442#comment-17610442
]
Alexander Lapin commented on IGNITE-17732:
--
[~v.pyatkov] LGTM
> Avoid locking indexes in transaction
>
>
> Key: IGNITE-17732
> URL: https://issues.apache.org/jira/browse/IGNITE-17732
> Project: Ignite
> Issue Type: Improvement
>Reporter: Vladislav Pyatkov
>Assignee: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3
>
> The transaction protocol requires lock all indexes (now it is a pk) in the
> same manner as it happens for data (locks on {_}RowIds{_}). But lock manager
> cannot resolve all problems in his case (in particular, waiter for _RowId_
> does not create when the lock on index have not got).
> As a temporary workaround for this disadvantage, supposed:
> # Create a NAL (Not a lock) lock mode, which is compatible for all another
> mode
> # All lock modes on PK should convert to NAL
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17732) Avoid locking indexes in transaction
[
https://issues.apache.org/jira/browse/IGNITE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-17732:
-
Reviewer: Alexander Lapin
> Avoid locking indexes in transaction
>
>
> Key: IGNITE-17732
> URL: https://issues.apache.org/jira/browse/IGNITE-17732
> Project: Ignite
> Issue Type: Improvement
>Reporter: Vladislav Pyatkov
>Assignee: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3
>
> The transaction protocol requires lock all indexes (now it is a pk) in the
> same manner as it happens for data (locks on {_}RowIds{_}). But lock manager
> cannot resolve all problems in his case (in particular, waiter for _RowId_
> does not create when the lock on index have not got).
> As a temporary workaround for this disadvantage, supposed:
> # Create a NAL (Not a lock) lock mode, which is compatible for all another
> mode
> # All lock modes on PK should convert to NAL
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17732) Avoid locking indexes in transaction
[
https://issues.apache.org/jira/browse/IGNITE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-17732:
-
Ignite Flags: (was: Docs Required,Release Notes Required)
> Avoid locking indexes in transaction
>
>
> Key: IGNITE-17732
> URL: https://issues.apache.org/jira/browse/IGNITE-17732
> Project: Ignite
> Issue Type: Improvement
>Reporter: Vladislav Pyatkov
>Assignee: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3
>
> The transaction protocol requires lock all indexes (now it is a pk) in the
> same manner as it happens for data (locks on {_}RowIds{_}). But lock manager
> cannot resolve all problems in his case (in particular, waiter for _RowId_
> does not create when the lock on index have not got).
> As a temporary workaround for this disadvantage, supposed:
> # Create a NAL (Not a lock) lock mode, which is compatible for all another
> mode
> # All lock modes on PK should convert to NAL
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-17771) Get rid of internal classes in API module.
Andrey Mashenkov created IGNITE-17771: - Summary: Get rid of internal classes in API module. Key: IGNITE-17771 URL: https://issues.apache.org/jira/browse/IGNITE-17771 Project: Ignite Issue Type: Improvement Reporter: Andrey Mashenkov Let's move 2 internal classes from ignite-api module. # org.apache.ignite.internal.sql.ResultSetImpl # org.apache.ignite.internal.sql.SqlColumnTypeConverter h3. Motivation. ResultSetImpl is just a wrapper over asynchronous resultset and was introduced to avoid unwanted code duplication as it is used in 2 modules (client and sql-engine). However, we may want to convert exception in different ways on clients and in embedded mode. So, having 2 similar implementations look acceptable. SqlColumnTypeConverter is utility class with a single static method, which converts SqlColumnType -> Java class. We can get rid of the class and add SqlColumnType.toJavaType() method instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17749) Remove unneeded zeroing of page in PageMemoryImpl.ClearSegmentRunnable
[
https://issues.apache.org/jira/browse/IGNITE-17749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610425#comment-17610425
]
Ivan Bessonov commented on IGNITE-17749:
[~ivandasch] Looks good to me, thank you for the improvement!
> Remove unneeded zeroing of page in PageMemoryImpl.ClearSegmentRunnable
> --
>
> Key: IGNITE-17749
> URL: https://issues.apache.org/jira/browse/IGNITE-17749
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Daschinsky
>Assignee: Ivan Daschinsky
>Priority: Major
> Labels: ise
>
> It seems that zeroing of page in `PageMemoryImpl.ClearSegmentRunnable`
> is not needed. Page, that is not borrowed from
> `PageMemoryImpl.Segment#loadedPages`, always either are zeroed explicitly or
> read from the persistence store.
> See `PageMemoryImpl.acquirePage` and `PageMemoryImpl.allocatePage`
> Zeroing of pages is a quite long operation in some cases, i.e. large memory
> region, see
> See excerpts from a series of threadumps fro a jfr recording, that was
> recorded on some server node.
> {code}
> TD_2022_08_23__21_57_40_1663679630023.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_57_40_1663679630023.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_57_40_1663679630023.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_58_10_1663679631047.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_58_10_1663679631047.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_58_10_1663679631047.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_58_40_1663679632677.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_58_40_1663679632677.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_58_40_1663679632677.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_59_10_1663679633562.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_59_10_1663679633562.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_59_10_1663679633562.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_59_40_1663679634489.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_59_40_1663679634489.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_59_40_1663679634489.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_00_10_1663679635411.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_00_10_1663679635411.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_00_10_1663679635411.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_00_40_1663679637003.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_00_40_1663679637003.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_00_40_1663679637003.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_01_10_1663679637840.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_01_10_1663679637840.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_01_10_1663679637840.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_01_40_1663679639483.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_01_40_1663679639483.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_01_40_1663679639483.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_02_10_1663679640384.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_02_10_1663679640384.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_02_10_1663679640384.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17749) Remove unneeded zeroing of page in PageMemoryImpl.ClearSegmentRunnable
[
https://issues.apache.org/jira/browse/IGNITE-17749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610423#comment-17610423
]
Ivan Daschinsky commented on IGNITE-17749:
--
[~ibessonov] Could you please review my patch?
> Remove unneeded zeroing of page in PageMemoryImpl.ClearSegmentRunnable
> --
>
> Key: IGNITE-17749
> URL: https://issues.apache.org/jira/browse/IGNITE-17749
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Daschinsky
>Assignee: Ivan Daschinsky
>Priority: Major
> Labels: ise
>
> It seems that zeroing of page in `PageMemoryImpl.ClearSegmentRunnable`
> is not needed. Page, that is not borrowed from
> `PageMemoryImpl.Segment#loadedPages`, always either are zeroed explicitly or
> read from the persistence store.
> See `PageMemoryImpl.acquirePage` and `PageMemoryImpl.allocatePage`
> Zeroing of pages is a quite long operation in some cases, i.e. large memory
> region, see
> See excerpts from a series of threadumps fro a jfr recording, that was
> recorded on some server node.
> {code}
> TD_2022_08_23__21_57_40_1663679630023.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_57_40_1663679630023.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_57_40_1663679630023.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_58_10_1663679631047.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_58_10_1663679631047.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_58_10_1663679631047.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_58_40_1663679632677.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_58_40_1663679632677.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_58_40_1663679632677.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_59_10_1663679633562.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_59_10_1663679633562.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_59_10_1663679633562.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_59_40_1663679634489.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_59_40_1663679634489.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_59_40_1663679634489.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_00_10_1663679635411.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_00_10_1663679635411.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_00_10_1663679635411.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_00_40_1663679637003.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_00_40_1663679637003.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_00_40_1663679637003.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_01_10_1663679637840.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_01_10_1663679637840.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_01_10_1663679637840.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_01_40_1663679639483.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_01_40_1663679639483.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_01_40_1663679639483.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_02_10_1663679640384.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_02_10_1663679640384.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_02_10_1663679640384.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17749) Remove unneeded zeroing of page in PageMemoryImpl.ClearSegmentRunnable
[
https://issues.apache.org/jira/browse/IGNITE-17749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610416#comment-17610416
]
Ignite TC Bot commented on IGNITE-17749:
{panel:title=Branch: [pull/10270/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/10270/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=6628033buildTypeId=IgniteTests24Java8_RunAll]
> Remove unneeded zeroing of page in PageMemoryImpl.ClearSegmentRunnable
> --
>
> Key: IGNITE-17749
> URL: https://issues.apache.org/jira/browse/IGNITE-17749
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Daschinsky
>Assignee: Ivan Daschinsky
>Priority: Major
> Labels: ise
>
> It seems that zeroing of page in `PageMemoryImpl.ClearSegmentRunnable`
> is not needed. Page, that is not borrowed from
> `PageMemoryImpl.Segment#loadedPages`, always either are zeroed explicitly or
> read from the persistence store.
> See `PageMemoryImpl.acquirePage` and `PageMemoryImpl.allocatePage`
> Zeroing of pages is a quite long operation in some cases, i.e. large memory
> region, see
> See excerpts from a series of threadumps fro a jfr recording, that was
> recorded on some server node.
> {code}
> TD_2022_08_23__21_57_40_1663679630023.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_57_40_1663679630023.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_57_40_1663679630023.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_58_10_1663679631047.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_58_10_1663679631047.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_58_10_1663679631047.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_58_40_1663679632677.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_58_40_1663679632677.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_58_40_1663679632677.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_59_10_1663679633562.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_59_10_1663679633562.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_59_10_1663679633562.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__21_59_40_1663679634489.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__21_59_40_1663679634489.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__21_59_40_1663679634489.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_00_10_1663679635411.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_00_10_1663679635411.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_00_10_1663679635411.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_00_40_1663679637003.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_00_40_1663679637003.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_00_40_1663679637003.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_01_10_1663679637840.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_01_10_1663679637840.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_01_10_1663679637840.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_01_40_1663679639483.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_01_40_1663679639483.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_01_40_1663679639483.log- at
> sun.misc.Unsafe.setMemory(Native Method)
> --
> TD_2022_08_23__22_02_10_1663679640384.log:"page-mem-op-#247087" #4407636
> prio=5 os_prio=0 tid=0x7ef984041000 nid=0x9639 runnable
> [0x7f8c732f3000]
> TD_2022_08_23__22_02_10_1663679640384.log- java.lang.Thread.State: RUNNABLE
> TD_2022_08_23__22_02_10_1663679640384.log- at
>
[jira] [Updated] (IGNITE-17688) Exception handling in transaction
[ https://issues.apache.org/jira/browse/IGNITE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Uttsel updated IGNITE-17688: --- Attachment: enlistInTx.png > Exception handling in transaction > - > > Key: IGNITE-17688 > URL: https://issues.apache.org/jira/browse/IGNITE-17688 > Project: Ignite > Issue Type: Improvement >Reporter: Sergey Uttsel >Assignee: Sergey Uttsel >Priority: Major > Labels: ignite-3, transaction3_rw > Attachments: enlistInTx.png > > > Need to describe exception handling in a transaction flow. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17688) Exception handling in transaction
[ https://issues.apache.org/jira/browse/IGNITE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Uttsel updated IGNITE-17688: --- Description: Need to describe exception handling in a transaction flow. (was: Need to describe handle exceptions in transaction) > Exception handling in transaction > - > > Key: IGNITE-17688 > URL: https://issues.apache.org/jira/browse/IGNITE-17688 > Project: Ignite > Issue Type: Improvement >Reporter: Sergey Uttsel >Assignee: Sergey Uttsel >Priority: Major > Labels: ignite-3, transaction3_rw > > Need to describe exception handling in a transaction flow. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17688) Exception handling in transaction
[ https://issues.apache.org/jira/browse/IGNITE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Uttsel updated IGNITE-17688: --- Description: Need to describe handle exceptions in transaction (was: Need to properly handle exceptions in transactions: # don't wrap TransactionException to another TransactionException # save error code, trace id.) > Exception handling in transaction > - > > Key: IGNITE-17688 > URL: https://issues.apache.org/jira/browse/IGNITE-17688 > Project: Ignite > Issue Type: Improvement >Reporter: Sergey Uttsel >Assignee: Sergey Uttsel >Priority: Major > Labels: ignite-3, transaction3_rw > > Need to describe handle exceptions in transaction -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17261) Implement a coordinator path write intent resolution logic for RO reads
[
https://issues.apache.org/jira/browse/IGNITE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Uttsel updated IGNITE-17261:
---
Description:
Because of lock-free nature, RO reads might interact with writeIntents, meaning
that such intents should be either evaluated as committed, aborted or pending.
In order to perform writeIntent resolution it's required to
* If PartitionReplicaListener read a write intent then it checks a local txn
state map for committed or aborted state - allow read if the state is committed
and commitTs <= readTs.
* If not possible, PartitionReplicaListener send TxStateReq to coordinator by
ReplicaService. - this initiates the {*}coordinator path{*}. Coordinator
address is fetched from the [txn state
map|https://docs.google.com/document/d/1PndaylEfK7CPUN7Kv9RYPKASN299s2qlbnMA5xIOFXo/edit#heading=h.wx3zf7jlf156].
* If a coordinator path was not able to resolve the intent, one of the
following has happened - the coordinator is dead or txn state is not available
in the cache. Calculate a commit partition and send the TxStateReq to its
primary replica - this initiates the {*}commit partition path{*}.
* Retry commit partition path until a success or timeout.
On receiving TxStateReq in ReplicaManager on the coordinator:
* ReplicaManager reads txn state map. If the local txn is finished, return the
response with the outcome: commit or abort. The txn state is stored in a local
cache (https://issues.apache.org/jira/browse/IGNITE-17638)
* If the local txn is finishing (txState == Finishing) waiting for finish
state replication, wait with timeout for outcome and return response with the
outcome: commit or abort. txState become Finishing in TxManager on creating
TxFinishReplicaRequest. TxManager has a txn state map. We can use future for
concurrency and atomic operations on txn state map.
* If the outcome is commit, additional timestamp check is required: a commit
timestamp must be <= readTs. If the condition is not held, the outcome is
changed to abort.
* If local txn is active (txState != [finishing, commit, abort]), adjust the
txn coordinator node HLC according to readTs to make sure the txn commit
timestamp is above the read timestamp. The read timestamp must be installed
before txn is started to commit, so commit timestamp is assigned after the read
timestamp.
* If txn state is not found in a local cache and txn is not active, return
NULL.
There's an open question about MvPartitionStorage api feature:
https://issues.apache.org/jira/browse/IGNITE-17627
was:
Because of lock-free nature, RO reads might interact with writeIntents, meaning
that such intents should be either evaluated as committed, aborted or pending.
In order to perform writeIntent resolution it's required to
* If PartitionReplicaListener read a write intent then it checks a local txn
state map for committed or aborted state - allow read if the state is committed
and commitTs <= readTs.
* If not possible, PartitionReplicaListener send TxStateReq to coordinator by
ReplicaService. - this initiates the {*}coordinator path{*}. Coordinator
address is fetched from the [txn state
map|https://docs.google.com/document/d/1PndaylEfK7CPUN7Kv9RYPKASN299s2qlbnMA5xIOFXo/edit#heading=h.wx3zf7jlf156].
* If a coordinator path was not able to resolve the intent, one of the
following has happened - the coordinator is dead or txn state is not available
in the cache. Calculate a commit partition and send the TxStateReq to its
primary replica - this initiates the {*}commit partition path{*}.
* Retry commit partition path until a success or timeout.
On receiving TxStateReq on the coordinator:
* PartitionReplicaListener read txn state map. If the local txn is finished,
return the response with the outcome: commit or abort. The txn state is stored
in a local cache (https://issues.apache.org/jira/browse/IGNITE-17638)
* If the local txn is finishing (txState == Finishing) waiting for finish
state replication, wait with timeout for outcome and return response with the
outcome: commit or abort. txState become Finishing in TxManager on creating
TxFinishReplicaRequest. TxManager has a txn state map.
* If the outcome is commit, additional timestamp check is required: a commit
timestamp must be <= readTs. If the condition is not held, the outcome is
changed to abort.
* If local txn is active (txState != [finishing, commit, abort]), adjust the
txn coordinator node HLC according to readTs to make sure the txn commit
timestamp is above the read timestamp. The read timestamp must be installed
before txn is started to commit, so commit timestamp is assigned after the read
timestamp. This must be achieved by some sort of concurrency control,
preferably non-blocking. In this case we must ignore the write intent, so the
outcome is to abort.
* If txn state is not found in a local cache and txn is not
[jira] [Updated] (IGNITE-17711) Change Binary Tuple Prefix format to allow comparison without deserialization
[ https://issues.apache.org/jira/browse/IGNITE-17711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-17711: --- Reviewer: Roman Puchkovskiy > Change Binary Tuple Prefix format to allow comparison without deserialization > - > > Key: IGNITE-17711 > URL: https://issues.apache.org/jira/browse/IGNITE-17711 > Project: Ignite > Issue Type: Improvement >Reporter: Aleksandr Polovtcev >Assignee: Aleksandr Polovtcev >Priority: Minor > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > BinaryTuplePrefix format dictates that the number of prefix elements are > appended to the end of the tuple. This is currently implemented as inserting > an extra int element which also implies an extra entry in the offset map and > non-deterministic size of this element (it can occupy from 1 to 4 bytes). > This was done for the sake of simplicity but it also has an implication: it > is not possible to simply compare it with a BinaryTuple byte-by-byte in order > to determine if a given prefix matches a given tuple. A better approach would > be to append the number of elements directly as the last two bytes (or four, > if necessary) of the tuple, bypassing the entry table. This way we can > discard the last two bytes and compare the prefix and the tuple on a per-byte > basis. -- This message was sent by Atlassian Jira (v8.20.10#820010)
