Re: Removing MVCC public API
+1 -- Regards, Konstantin Orlov Software Engineer, St. Petersburg +7 (921) 445-65-75 https://www.gridgain.com Powered by Apache® Ignite™
[VOTE] Release Apache Ignite 2.9.1 RC1
Dear Community, I have uploaded release candidate to https://dist.apache.org/repos/dist/dev/ignite/2.9.1-rc1/ https://dist.apache.org/repos/dist/dev/ignite/packages_2.9.1-rc1/ The following staging can be used for testing: https://repository.apache.org/content/repositories/orgapacheignite-1490/ Tag name is 2.9.1-rc1: https://gitbox.apache.org/repos/asf?p=ignite.git;a=tag;h=refs/tags/2.9.1-rc1 2.9.1 changes: * Added support to graceful shutdown for ZookeeperDiscoverySpi * Added System view for binary metadata * Added System view for metastorage items * Added RebalancingPartitionsTotal metrics * Improved checkpoint concurrent behaviour * Fixed critical system error when unregister a JMX bean * Fixed IgniteCache#isClosed() returns false on server node even if the cache had been closed before * Fixed inability to eagerly remove expired cache entries * Fixed partial index rebuild issue in case indexed cache contains different datatypes * Fixed local metastorage system view error if unmarshallable values present * Fixed deadlock between grid-timeout-worker and a thread opening a communication connection * Fixed deadlock in IgniteServiceProcessor * Fixed issue when rebalance future might hang in no final state though all partitions are owned * Fixed issue when scan query fails with an assertion error: Unexpected row key * Fixed issue with archiving and enabled wal compaction setting on server restart * Fixed NPE during Cassandra Store initialization with PRIMITIVE strategy * Fixed synchronization problems when different classloaders are used for deployment of same class * Fixed exception on SQL caches when client reconnect * Fixed deadlock on multiple cache delete * Fixed NPE in IgniteServiceProcessor when destroying a cache * Fixed issue when DurableBackgroundTask can abandon incomplete task * Fixed issue related to cache interceptor deserialization on client nodes * Fixed issue when control.sh doesn't start if JMX port was set * Fixed issue when ZookeeperDiscoverySpiMBeanImpl#getCoordinator can return invalid node as coordinator * Fixed issue when valid blocking section in GridNioWorker and GridNioClientWorker leads to false positive blocking thread detection * Fixed several logging issues * Fixed issue when exchange worker, waiting for new task from queue, considered as blocked * Fixed incorrect topology snapshot logger output about coordinator change * Fixed slowdown during node initialization * Fixed incorrect usage of Class.isAssignableFrom in SystemViewLocal and SystemViewMBean classes * Fixed several concurrent metrics issues * Fixed issue related to incorrect work of predicates (< and >) in where clause with compound primary key * Removed unnecessary dependency to curator-client from and improved ZookeeperDiscoverySpi * Removed unnecessary failure trace in IgnitionEx * Fixed issue when thin client connect/disconnect during topology update may lead to partition divergence in ignite-sys-cache * Fixed issue when thin client silently closes channel after inactivity * Fixed unsupported protocol version exception when getting cache configuration from Java thin client * Fixed thin driver reports incorrect property names * Updated JDBC metadata to match actual capabilities * Improved slow Enum serialization * Fixed issue when deserializing IBinaryObject containing an IBinaryObject field fails * Fixed issue TransactionImpl finalizer can crash the process * Fixed child processes become zombies when persistence is used with direct-io on Linux * Added Windows support to CMake build system * Fixed issue when odbc-example loses values if run with 1 additional node RELEASE NOTES: https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob;f=RELEASE_NOTES.txt;hb=ignite-2.9.1 Complete list of resolved issues: https://issues.apache.org/jira/browse/IGNITE-13770?jql=project%20%3D%20IGNITE%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.9.1 DEVNOTES https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.9.1 The vote is formal, see voting guidelines https://www.apache.org/foundation/voting.html +1 - to accept Apache Ignite 2.9.1-rc1 0 - don't care either way -1 - DO NOT accept Apache Ignite Ignite 2.9.1-rc1 (explain why) See notes on how to verify release here https://www.apache.org/info/verification.html and https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification This vote will be open for at least 3 days till Fri Dec 18, 10:30 am UTC. https://www.timeanddate.com/countdown/generic?iso=20201218T1030&p0=166&font=cursive
[jira] [Created] (IGNITE-13854) Add documentation for the cluster performance profiling tool
Amelchev Nikita created IGNITE-13854: Summary: Add documentation for the cluster performance profiling tool Key: IGNITE-13854 URL: https://issues.apache.org/jira/browse/IGNITE-13854 Project: Ignite Issue Type: Sub-task Reporter: Amelchev Nikita Assignee: Amelchev Nikita Add documentation for the cluster performance profiling tool. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13853) Add APIs, examples, language selector to the docs menu
Denis A. Magda created IGNITE-13853: --- Summary: Add APIs, examples, language selector to the docs menu Key: IGNITE-13853 URL: https://issues.apache.org/jira/browse/IGNITE-13853 Project: Ignite Issue Type: Improvement Components: documentation, website Reporter: Denis A. Magda Fix For: 2.9.1 There is user feedback that suggests we need to rework our navigation menus of the website and docs to improve the discoverability of APIs and examples. Let's do the following. *Ignite Website Top-Level Navigation Menu ([https://ignite.apache.org)|https://ignite.apache.org)/]* * Add the "Docs" menu item to the menu placing it before the "Features" item. The "Docs" _is a single link_ (no dropdown menu) that directs to the root of the latest technical documentation: [https://ignite.apache.org/docs/latest/] * Rename "Resources" to "Learn More". Remove "Documentation", "Documentation (Chinese)" and "APIs" from this menu section. * Remove "Events" from the top-level menu. Add the items/links from the menu to the "Community" *Technical Documentation Top-Level Navigation Menu ([https://ignite.apache.org/docs/latest/])* * Remove the Github icon from the menu * Add "APIs" after the "version" selector. The menu lists "JavaDocs", "C#/.NET", "C++", "Scala". The "version" selector defines what version of an API is to be opened. For instance, if a user chooses version "2.9.1" (to be released soon) then "JavaDocs" of that version is to be opened. If instead, the user wants to see the docs and APIs for "2.9.0", he just needs to selects that version from the version menu. * Add "Examples" after the "APIs". Add the following menu items: ** "Java": [https://github.com/apache/ignite/tree/master/examples] ** "C#/.NET": [https://github.com/apache/ignite/tree/master/modules/platforms/dotnet/examples] ** "C++": [https://github.com/apache/ignite/tree/master/modules/platforms/cpp/examples] ** "Python": [https://github.com/apache/ignite/tree/master/modules/platforms/python/examples] ** "Node.JS": [https://github.com/apache/ignite/tree/master/modules/platforms/nodejs/examples] ** "PHP": [https://github.com/apache/ignite/tree/master/modules/platforms/php/examples] * Add a language selector drop-down list after the "search" text box. See how it's implemented here: [https://www.confluent.io.|https://www.confluent.io./] We'll have just two menu items there: ** "English": points out to the root of the main docs [https://ignite.apache.org/docs/latest/] ** "Chinese": https://www.ignite-service.cn/doc/java/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13852) Ascii tables for new unified cli tool
Kirill Gusakov created IGNITE-13852: --- Summary: Ascii tables for new unified cli tool Key: IGNITE-13852 URL: https://issues.apache.org/jira/browse/IGNITE-13852 Project: Ignite Issue Type: Task Reporter: Kirill Gusakov Implement Ascii tables with support of Ansi symbols and custom borders -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Ignite 3.0 development approach
So just one bit to consider... Are there features that you need to use in these newer versions of java? Or are we just updating to stay current? The reason I ask is that there are still lots of people in an enterprise space that are beholden to having to support legacy JAVAEE supported applications on Websphere, Weblogic, Redhat, etc... and the organizations that use those platforms are slow to move... Most of them are still on Java8 So as a platform I think a strong consideration needs to be towards having the broadest possible support profile until it becomes an impediment to doing things that the platform needs. So far I haven't seen huge things in the newer versions of java that are must haves ( a few exceptions are things that would be really nice to take advantage of ). I think that apache commons has taken the right approach by staying on java 8 giving the largest possible user base. Even standardizing on java 11 would have to make us reconsider Ignite as the platform we are using, we are not so invested in it as of now, even though we have big plans to leverage it. Just because we aren't sure when we are going to be able to upgrade from java8. It has support through 2022, so I imagine that is when we will be discussing that. Regards ~Adam Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline Technologies Office: 603-501-6446 | Mobile: 603-570-8418 www.bottomline.com On 11/24/20, 7:38 AM, "Alexey Zinoviev" wrote: Java 15 is better, VarHandles, ForeignMemory access and so on. In both cases, I support the Java version 11 and higher for the development! вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov : > Let's add maven plugins for sanity checks at the early stage. > I've created a ticket for this [1]. > > Also, I've found initial pom.xml has a target version Java 8. > Do we intend to move to Java 11 (or may be next LTS) and drop Java 8 in > Ignite 3.0? > > [1] https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$ > > On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > Folks, > > > > I went ahead and created the repository [1]. I also configured a TeamCity > > project [2] that runs all available JUnit tests on every PR creation or > > update. It also sends the status update to GitHub so that it's reflected > in > > the PR itself so that we can do merges directly from GitHub. Basic steps > to > > make a change are described on the Wiki page [3]. > > > > Let me know if you have any questions. > > > > [1] https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$ > > [2] https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$ > > [3] > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$ > > > > -Val > > > > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko < > > valentin.kuliche...@gmail.com> wrote: > > > > > Thanks, guys. It looks like we are much closer to the consensus now. I > > > totally on board with the plan, but I would also like to address the > > > short-term needs. As I've already mentioned earlier, there are several > > > active IEPs, but we still don't have even a preliminary technical > process > > > for working on these IEPs. I believe this might be frustrating for the > > > folks who would like to commit code. > > > > > > The scope we agreed on is quite big, and it will surely take > significant > > > time to implement all the changes and stabilize them. Therefore, it's > > clear > > > to me that we will have to maintain 2.x and 3.x in parallel for quite > > some > > > time - this needs to be addressed somehow. I'm convinced that having a > > > separate repo is the ONLY way to do that, and so far, I haven't heard > any > > > clear alternatives or reasons why we shouldn't do this. > > > > > > That said, I'm inclined to proceed with this in the next few days - I > > will > > > create a repo and describe the process (which we, of course, can > discuss > > > and modify going forward). Let's, at the very least, try and see where > it > > > leads us. > > > > > > If someone has any concrete alternative options on how to we can > maintain > > > two major versions in parallel, let's have another voice discussion > this > > > F
[GitHub] [ignite-3] vveider closed pull request #7: Apache Ignite 3: RPM build
vveider closed pull request #7: URL: https://github.com/apache/ignite-3/pull/7 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [ignite-3] vveider opened a new pull request #7: Apache Ignite 3: RPM build
vveider opened a new pull request #7: URL: https://github.com/apache/ignite-3/pull/7 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (IGNITE-13851) Document persistence forward compatibility policy
Ilya Kasnacheev created IGNITE-13851: Summary: Document persistence forward compatibility policy Key: IGNITE-13851 URL: https://issues.apache.org/jira/browse/IGNITE-13851 Project: Ignite Issue Type: Improvement Components: documentation Reporter: Ilya Kasnacheev Fix For: 2.10 We have a policy that you can upgrade from 2.x to 2.y, y > x, while keeping your persistence files. Let's document that fact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Tool for performance statistics reports
Guys, I have filed the ticket [1] to improve the TracingSpi interface and further reusing it for cluster profiling. I'll assign it to me and take it to work after profiling merge if nobody minds. [1] https://issues.apache.org/jira/browse/IGNITE-13850 пн, 14 дек. 2020 г. в 13:38, Alexander Lapin : > > Ok from my side. > Few more details about tracing spi updates, based on mentioned above > discussion with Nikolay and Nikita. > > Tracing provides enough data for a performance profiling tool, actually > only root spans are required. However, according to Nikinta, > root-span-tracing has a 7-8% performance drop in comparison to 1-2% > performance drop of the performance profiling tool. It's the main reason to > have given tool as is right now. In order to reuse TracingSPI for a > profiling tool internals, few modifications should be made to increase > tracing performance: > >- Add support for non-strings tags and log points: primitives, etc. >- Add ability to postpone adding span tags and log points to the very >and of span tree creation. >- Probably some sort of tags caching could also help. > > Best regards, > Alexander > > пн, 14 дек. 2020 г. в 12:48, Nikolay Izhikov : > > > Hello, Igniters. > > > > We discussed this feature privately with Alexander and Nikita. > > Here are the results we want to share with the community: > > > > 0. In the end, both, performance statistic tool and tracing should use the > > same API. > > 1. We should improve the Tracing API, so it able to be used for gathering > > information about all operations without a significant performance drop. > > > > I propose to go as follows: > > > > 1. Merge current PR as is after final review. My intention is to provide a > > tool for users that can be used in the real-world production environment. > > 2. Improve the Tracing API. > > 3. Combine both tools under the same API. > > > > > 14 дек. 2020 г., в 10:42, Alexander Lapin > > написал(а): > > > > > > Hello Igniters, > > > > > > Because the tracing causes performance drop 52% [4] and can not be > > >> used for collecting statistics about all queries in production > > >> deployments. The performance drop of the profiling tool is less than > > >> 2% and it can be used in production. I have benchmarked the tracing > > >> and got the results: > > >> > > >> -2% when configured OpenCensusTracingSpi and all scopes disabled > > >> -52% for TX scope (IgnitePutTxBenchmark) > > >> -58% for SQL scope (IgniteSqlQueryBenchmark) > > >> > > >> Such a performance drop is significant to not use the tracing in > > >> production. > > >> > > > We've rerun tracing benchmarks based on more realistic scenarios and got > > a > > > 10-15% performance drop in case of sampling-rate 1 (all transactions were > > > traced). More realistic scenarios means that we don't test tracing > > > performance if the system is in overdraft state but add some sort of > > micro > > > throttling (1 millisecond) between operations, transactions in our case. > > > *IgnitePutTxBenchmark* > > > > > > Green: Case 1: NoopTracingSpi > > > > > > Blue: Case 2: OpenCensusTracingSpi (disabled) > > > > > > Red: Case 3: OpenCensusTracingSpi, --scope TX --sampling-rate 0.1 > > > > > > Black: Case 5: *ControlCenter* + OpenCensusTracingSpi, --scope TX > > > --sampling-rate 0.1 > > > > > > Violet: Case 4: OpenCensusTracingSpi, --scope TX --sampling-rate 1 > > > Yellow: Case 6: ControlCenter + OpenCensusTracingSpi, --scope TX > > > --sampling-rate > > > > > > I have considered the possibility to reuse the tracing API. If > > >> statistics collecting will be implemented with the TracingSpi then we > > >> get a performance drop due to: > > >> - Transferring tracing context over the network. > > >> - Using ThreadLocal for spans > > >> - Converting primitives and objects to string and vice versa. (API > > >> supports only String-based tags and values) > > >> - Generating span objects > > >> > > > @Nikita Amelchev Could you please share numbers? > > > > > > Best regards, > > > Alexander > > > > > > пн, 7 дек. 2020 г. в 17:24, Nikolay Izhikov : > > > > > >> Hello, Nikita. > > >> > > >> Makes sense. > > >> > > >> I will take a look. > > >> > > >>> 7 дек. 2020 г., в 15:29, Nikita Amelchev > > >> написал(а): > > >>> > > >>> Hello, Igniters. > > >>> > > >>> I have implemented the profiling tool [1, 2]. It writes duration and > > >>> other parameters of user operations (scan, SQL query, transactions, > > >>> tasks, jobs, CQ, etc) to a local file. This info can be used in > > >>> various cases. The main goal is to build the performance report to > > >>> analyze the count and duration of user queries [3]. > > >>> > > >>> We already have the tracing with similar functionality but I think > > >>> Ignite should have both tools - tracing and profiling. > > >>> > > >>> Because the tracing causes performance drop 52% [4] and can not be > > >>> used for collecting statistics about all queries in production > > >>> deployments. The performance drop of t
[jira] [Created] (IGNITE-13850) Improve the TracingSpi interface
Amelchev Nikita created IGNITE-13850: Summary: Improve the TracingSpi interface Key: IGNITE-13850 URL: https://issues.apache.org/jira/browse/IGNITE-13850 Project: Ignite Issue Type: Improvement Reporter: Amelchev Nikita Assignee: Amelchev Nikita Need to investigate and improve the TracingSpi interface: 1. Add support for non-strings tags and log points: primitives, arrays etc 2. Add ability to postpone adding span tags and log points 3. Investigate ability to cache some tags -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Tool for performance statistics reports
Ok from my side. Few more details about tracing spi updates, based on mentioned above discussion with Nikolay and Nikita. Tracing provides enough data for a performance profiling tool, actually only root spans are required. However, according to Nikinta, root-span-tracing has a 7-8% performance drop in comparison to 1-2% performance drop of the performance profiling tool. It's the main reason to have given tool as is right now. In order to reuse TracingSPI for a profiling tool internals, few modifications should be made to increase tracing performance: - Add support for non-strings tags and log points: primitives, etc. - Add ability to postpone adding span tags and log points to the very and of span tree creation. - Probably some sort of tags caching could also help. Best regards, Alexander пн, 14 дек. 2020 г. в 12:48, Nikolay Izhikov : > Hello, Igniters. > > We discussed this feature privately with Alexander and Nikita. > Here are the results we want to share with the community: > > 0. In the end, both, performance statistic tool and tracing should use the > same API. > 1. We should improve the Tracing API, so it able to be used for gathering > information about all operations without a significant performance drop. > > I propose to go as follows: > > 1. Merge current PR as is after final review. My intention is to provide a > tool for users that can be used in the real-world production environment. > 2. Improve the Tracing API. > 3. Combine both tools under the same API. > > > 14 дек. 2020 г., в 10:42, Alexander Lapin > написал(а): > > > > Hello Igniters, > > > > Because the tracing causes performance drop 52% [4] and can not be > >> used for collecting statistics about all queries in production > >> deployments. The performance drop of the profiling tool is less than > >> 2% and it can be used in production. I have benchmarked the tracing > >> and got the results: > >> > >> -2% when configured OpenCensusTracingSpi and all scopes disabled > >> -52% for TX scope (IgnitePutTxBenchmark) > >> -58% for SQL scope (IgniteSqlQueryBenchmark) > >> > >> Such a performance drop is significant to not use the tracing in > >> production. > >> > > We've rerun tracing benchmarks based on more realistic scenarios and got > a > > 10-15% performance drop in case of sampling-rate 1 (all transactions were > > traced). More realistic scenarios means that we don't test tracing > > performance if the system is in overdraft state but add some sort of > micro > > throttling (1 millisecond) between operations, transactions in our case. > > *IgnitePutTxBenchmark* > > > > Green: Case 1: NoopTracingSpi > > > > Blue: Case 2: OpenCensusTracingSpi (disabled) > > > > Red: Case 3: OpenCensusTracingSpi, --scope TX --sampling-rate 0.1 > > > > Black: Case 5: *ControlCenter* + OpenCensusTracingSpi, --scope TX > > --sampling-rate 0.1 > > > > Violet: Case 4: OpenCensusTracingSpi, --scope TX --sampling-rate 1 > > Yellow: Case 6: ControlCenter + OpenCensusTracingSpi, --scope TX > > --sampling-rate > > > > I have considered the possibility to reuse the tracing API. If > >> statistics collecting will be implemented with the TracingSpi then we > >> get a performance drop due to: > >> - Transferring tracing context over the network. > >> - Using ThreadLocal for spans > >> - Converting primitives and objects to string and vice versa. (API > >> supports only String-based tags and values) > >> - Generating span objects > >> > > @Nikita Amelchev Could you please share numbers? > > > > Best regards, > > Alexander > > > > пн, 7 дек. 2020 г. в 17:24, Nikolay Izhikov : > > > >> Hello, Nikita. > >> > >> Makes sense. > >> > >> I will take a look. > >> > >>> 7 дек. 2020 г., в 15:29, Nikita Amelchev > >> написал(а): > >>> > >>> Hello, Igniters. > >>> > >>> I have implemented the profiling tool [1, 2]. It writes duration and > >>> other parameters of user operations (scan, SQL query, transactions, > >>> tasks, jobs, CQ, etc) to a local file. This info can be used in > >>> various cases. The main goal is to build the performance report to > >>> analyze the count and duration of user queries [3]. > >>> > >>> We already have the tracing with similar functionality but I think > >>> Ignite should have both tools - tracing and profiling. > >>> > >>> Because the tracing causes performance drop 52% [4] and can not be > >>> used for collecting statistics about all queries in production > >>> deployments. The performance drop of the profiling tool is less than > >>> 2% and it can be used in production. I have benchmarked the tracing > >>> and got the results: > >>> > >>> -2% when configured OpenCensusTracingSpi and all scopes disabled > >>> -52% for TX scope (IgnitePutTxBenchmark) > >>> -58% for SQL scope (IgniteSqlQueryBenchmark) > >>> > >>> Such a performance drop is significant to not use the tracing in > >> production. > >>> > >>> I have considered the possibility to reuse the tracing API. If > >>> statistics collecting will be implemented with the
[jira] [Created] (IGNITE-13849) Calcite Integration. Add unit tests for all IgniteRel nodes serialization / deserialization
Taras Ledkov created IGNITE-13849: - Summary: Calcite Integration. Add unit tests for all IgniteRel nodes serialization / deserialization Key: IGNITE-13849 URL: https://issues.apache.org/jira/browse/IGNITE-13849 Project: Ignite Issue Type: Bug Components: sql Reporter: Taras Ledkov Assignee: Taras Ledkov We have to check serialization for each node (child classes of the IgniteRel). -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Tool for performance statistics reports
Hello, Igniters. We discussed this feature privately with Alexander and Nikita. Here are the results we want to share with the community: 0. In the end, both, performance statistic tool and tracing should use the same API. 1. We should improve the Tracing API, so it able to be used for gathering information about all operations without a significant performance drop. I propose to go as follows: 1. Merge current PR as is after final review. My intention is to provide a tool for users that can be used in the real-world production environment. 2. Improve the Tracing API. 3. Combine both tools under the same API. > 14 дек. 2020 г., в 10:42, Alexander Lapin написал(а): > > Hello Igniters, > > Because the tracing causes performance drop 52% [4] and can not be >> used for collecting statistics about all queries in production >> deployments. The performance drop of the profiling tool is less than >> 2% and it can be used in production. I have benchmarked the tracing >> and got the results: >> >> -2% when configured OpenCensusTracingSpi and all scopes disabled >> -52% for TX scope (IgnitePutTxBenchmark) >> -58% for SQL scope (IgniteSqlQueryBenchmark) >> >> Such a performance drop is significant to not use the tracing in >> production. >> > We've rerun tracing benchmarks based on more realistic scenarios and got a > 10-15% performance drop in case of sampling-rate 1 (all transactions were > traced). More realistic scenarios means that we don't test tracing > performance if the system is in overdraft state but add some sort of micro > throttling (1 millisecond) between operations, transactions in our case. > *IgnitePutTxBenchmark* > > Green: Case 1: NoopTracingSpi > > Blue: Case 2: OpenCensusTracingSpi (disabled) > > Red: Case 3: OpenCensusTracingSpi, --scope TX --sampling-rate 0.1 > > Black: Case 5: *ControlCenter* + OpenCensusTracingSpi, --scope TX > --sampling-rate 0.1 > > Violet: Case 4: OpenCensusTracingSpi, --scope TX --sampling-rate 1 > Yellow: Case 6: ControlCenter + OpenCensusTracingSpi, --scope TX > --sampling-rate > > I have considered the possibility to reuse the tracing API. If >> statistics collecting will be implemented with the TracingSpi then we >> get a performance drop due to: >> - Transferring tracing context over the network. >> - Using ThreadLocal for spans >> - Converting primitives and objects to string and vice versa. (API >> supports only String-based tags and values) >> - Generating span objects >> > @Nikita Amelchev Could you please share numbers? > > Best regards, > Alexander > > пн, 7 дек. 2020 г. в 17:24, Nikolay Izhikov : > >> Hello, Nikita. >> >> Makes sense. >> >> I will take a look. >> >>> 7 дек. 2020 г., в 15:29, Nikita Amelchev >> написал(а): >>> >>> Hello, Igniters. >>> >>> I have implemented the profiling tool [1, 2]. It writes duration and >>> other parameters of user operations (scan, SQL query, transactions, >>> tasks, jobs, CQ, etc) to a local file. This info can be used in >>> various cases. The main goal is to build the performance report to >>> analyze the count and duration of user queries [3]. >>> >>> We already have the tracing with similar functionality but I think >>> Ignite should have both tools - tracing and profiling. >>> >>> Because the tracing causes performance drop 52% [4] and can not be >>> used for collecting statistics about all queries in production >>> deployments. The performance drop of the profiling tool is less than >>> 2% and it can be used in production. I have benchmarked the tracing >>> and got the results: >>> >>> -2% when configured OpenCensusTracingSpi and all scopes disabled >>> -52% for TX scope (IgnitePutTxBenchmark) >>> -58% for SQL scope (IgniteSqlQueryBenchmark) >>> >>> Such a performance drop is significant to not use the tracing in >> production. >>> >>> I have considered the possibility to reuse the tracing API. If >>> statistics collecting will be implemented with the TracingSpi then we >>> get a performance drop due to: >>> - Transferring tracing context over the network. >>> - Using ThreadLocal for spans >>> - Converting primitives and objects to string and vice versa. (API >>> supports only String-based tags and values) >>> - Generating span objects >>> >>> I have benchmarked implementations on the yardstick’s >>> IgniteGetBenchmark. The tracing context transferring over the network >>> was disabled. The results: >>> - Tracing API implementation - 8% performance drop. >>> - Proposed implementation - 2% performance drop. >>> >>> I think this is a significant drop and implementation with reuse >>> tracing API should not be used. The cluster profiling should have as >>> little performance drop as possible to be used in production. The >>> tracing will be used for the detailed investigation. >>> >>> WDYT? >>> >>> The tool is ready to be reviewed [3, 5]. >>> >>> [1] https://issues.apache.org/jira/browse/IGNITE-12666 >>> [2] >> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performan
[jira] [Created] (IGNITE-13848) Premature update SegmentReservationStorage#minReserveIdx during truncate of segments
Kirill Tkalenko created IGNITE-13848: Summary: Premature update SegmentReservationStorage#minReserveIdx during truncate of segments Key: IGNITE-13848 URL: https://issues.apache.org/jira/browse/IGNITE-13848 Project: Ignite Issue Type: Bug Components: persistence Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.10 It was found premature *SegmentReservationStorage#minReserveIdx* update in *FileWriteAheadLogManager#truncate*. Which creates the wrong state of the segments in the archive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
