Re: Why WAL archives enabled by default?
Dmitriy, as far as I understand, Denis have adviced user to "disable" archiving by setting wal and wal archive path to the same value. I tried to explain that this measure doesn't prevent from storing wal segments needed for recovery but bring additional performance penalty. > In older versions of Apache Ignite, WAL archive could contain valid records needed for recovery. If something was changed since then, my comment may be not valid. Nothing changed since that time. ср, 11 нояб. 2020 г., 03:31 Dmitriy Pavlov : > In older versions of Apache Ignite, WAL archive could contain valid records > needed for recovery. If something was changed since then, my comment may be > not valid. > > We've discussed that before, that naming this directory as 'archive' was > not the best possible option. The archive is often considered by users as > something not needed and sometimes it was deleted. > > See also page related to internals and directory structure: > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStoreunderthehood-WALstructure > > So infinite storage of archive is definitely not necessary for vanilla > open-source version, but archive itself is needed. > > Sincerely, > Dmitriy Pavlov > > ср, 11 нояб. 2020 г. в 01:21, Raymond Wilson : > > > Isn't the discussion here related to the WAL archive? If you disable that > > don't you still have the WAL containing un-checkpointed changes? > > > > On Wed, Nov 11, 2020 at 11:01 AM Dmitriy Pavlov > > wrote: > > > > > Hi Denis, > > > > > > the short answer here, Apache Ignite guarantees ACID, and for > > D-Durability > > > it is required to save all changes in some WAL/Redo Log to have a safe > > way > > > to recover from any hardware failures/disk outage. > > > > > > Should the user disable WAL, he/she could potentially lose durability. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > вт, 10 нояб. 2020 г. в 09:57, ткаленко кирилл : > > > > > > > Hello guys again! > > > > > > > > Does anyone know why we are doing any calculation here > > > > IgniteUtils#adjustedWalHistorySize at all? > > > > Would it be easier to always take the > > > > DataStorageConfiguration#maxWalArchiveSize? It seems that the user > can > > > > easily do this himself by changing the value by 1 byte. > > > > > > > > 06.11.2020, 13:56, "Ivan Daschinsky" : > > > > > Alex, thanks for pointing that out. Shame that I missed it. > > > > > > > > > > пт, 6 нояб. 2020 г. в 13:45, Alex Plehanov < > plehanov.a...@gmail.com > > >: > > > > > > > > > >> Guys, > > > > >> > > > > >> We already have > > > FileWriteAheadLogManager#maxSegCountWithoutCheckpoint. > > > > >> Checkpoint triggered if there are too many WAL segments without > > > > checkpoint. > > > > >> Looks like you are talking about this feature. > > > > >> > > > > >> пт, 6 нояб. 2020 г. в 13:21, Ivan Daschinsky < > ivanda...@gmail.com > > >: > > > > >> > > > > >> > Kirill and I discussed privately proposed approach. As far as I > > > > >> understand, > > > > >> > Kirill suggests to implement some > > > > >> > heuristic to do a force checkpoint in some cases if user by > > mistake > > > > >> > misconfigured cluster in order to preserve > > > > >> > requested size of WAL archive. > > > > >> > Currently, as for me, this approach is questionable, because it > > can > > > > cause > > > > >> > some performance problems. But as an option, > > > > >> > it can be used and should be switchable. > > > > >> > > > > > >> > пт, 6 нояб. 2020 г. в 12:36, Ivan Daschinsky < > > ivanda...@gmail.com > > > >: > > > > >> > > > > > >> > > Kirill, how your approach will help if user tuned a cluster > to > > do > > > > >> > > checkpoints rarely under load? > > > > >> > > No way. > > > > >> > > > > > > >> > > пт, 6 нояб. 2020 г. в 12:19, ткаленко кирилл < > > > tkalkir...@yandex.ru > > > > >: > > > > >> > > > > > > >> > >> Ivan, I agree with you that the archive is primarily about > > > > >> optimization. > > > > >> > >> > > > > >> > >> If the size of the archive is critical for the user, we have > > no > > > > >> > >> protection against this, we can always go beyond this limit. > > > > >> > >> Thus, the user needs to remember this and configure it in > some > > > > way. > > > > >> > >> > > > > >> > >> I suggest not to exceed this limit and give the expected > > > behavior > > > > for > > > > >> > the > > > > >> > >> user. At the same time, the segments needed for recovery > will > > > > remain > > > > >> and > > > > >> > >> there will be no data loss. > > > > >> > >> > > > > >> > >> 06.11.2020, 11:29, "Ivan Daschinsky" : > > > > >> > >> > Guys, fisrt of all, archiving is not for PITR at all, this > > is > > > > >> > >> optimization. > > > > >> > >> > If we disable archiving, every rollover we need to create > > new > > > > file. > > > > >> If > > > > >> > >> we > > > > >> > >> > enable archiving, we reserve 10 (by default) segments > filled > > > > with > > > > >>
Re: [DISCUSS] Ignite 3.0 development approach
Maxim, 2.x and 3.x will have to coexist for some time - I don't see how we can avoid this considering the set of proposed changes. That said, we effectively will need to have two "masters" - one for each major version. Master for 3.x can technically be a branch in the existing repo, but having a separate repo seems cleaner, simply because it will not be a "branch" in the traditional sense. Note that the new repo will still be under the Apache org, with the same set of committers, managed by the community, etc. All the development happening for 3.0 must follow the rules that we currently have (if anything, it's an opportunity to improve those rules). As I said during the call on Friday, I strongly believe that if there is a transparency issue, it will exist regardless of the approach we choose for 3.0. If community members develop without IEPs or public discussions, this will happen for both 2.x and 3.x unless we address this separately. I don't see how this is related to Alexey's suggestion, which targets *technical* issues with the product more than anything else. This a way to achieve better modularity, introduce better coverage with unit tests, reduce conflicts during development, etc. Coming back to transparency, let's identify the issues and fix them. It probably makes sense to have a separate discussion on this topic. -Val On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov wrote: > Sergey, > > > Your summary makes sense to me. > > However, how we come up from *Development transparency* to *create a > separate public repository dedicated for 3.0*? For me *development > transparency* is about making changes in the master branch. These > changes will definitely be seen by all the Ignite developers. > > A dedicated public repository is technically public and visible for > everyone, but it allows development without IEPs, without public > discussion (since all the code changes are not related to the master > branch) it also allows a large number of assumptions and deviations > (like code-style violations). It also not about *development > transparency* since developers which are working on 3.0 is only a > subset of all Ignite developers which may continue working on 2.x. For > me, this would be a huge step backwards. > > Ignite veterans should remember how long the branch stabilization took > for the 2.x version with the PDS. > > > I think each breaking change should be passed through the master branch. > > On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov > wrote: > > > > Makes sense to me. > > > > вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov >: > > > > > Igniters, > > > > > > I thought over Friday meeting ideas and concerns and summarized them in > > > these three points: > > > > > > > > >1. *Components design unification approach.* New proposed components > > >will be developed by different contributors, but they need to be > unified > > >and should integrate with each other easily. To ensure that I > suggest > > >calling an architecture group that will create design guidelines > for all > > >components and high-level overview of overall architecture. How > code is > > >split into components, what are component boundaries, how component > > >lifecycle works and what are its interfaces - all these and other > > > questions > > >should be covered. > > > > > >2. *Scope management.* Apache 3.0 should be implemented within a > > >reasonable time, so we need some procedure to decide whether a > > > particular > > >feature should be dropped from the scope of 3.0 and postponed to 3.1 > > >release. To do so I suggest to range all features by two parameters: > > >criticality for 3.0 and amount of breaking changes. 3.0 scope should > > >include features of high criticality AND features with a big amount > of > > >breaking changes. All other features can be made optional. > > > > > >3. *Development transparency.* Development of all components should > be > > >made as transparent for everyone as possible. Any contributor > should be > > >able to look over any component at any stage of development. To > achieve > > >this I suggest to create a separate public repository dedicated for > 3.0 > > >development. It will make the code available for everyone but when > > >development of 3.0 is done we won't loose any stars of our current > > >repository as we merge dev repo into main one and drop dev. > > > > > > Do these ideas make sense to you? Are there any concerns not covered by > > > these suggestions? > > > > > > On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova < > romanova.ks@gmail.com > > > > > > > wrote: > > > > > > > Here are the slides from Alexey Goncharuk. Let's think this over and > > > > continue on Monday: > > > > > > > > > > > > https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf > > > > > > > > чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov : > > > > > > > > > Folks, > > >
Re: Why WAL archives enabled by default?
In older versions of Apache Ignite, WAL archive could contain valid records needed for recovery. If something was changed since then, my comment may be not valid. We've discussed that before, that naming this directory as 'archive' was not the best possible option. The archive is often considered by users as something not needed and sometimes it was deleted. See also page related to internals and directory structure: https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStoreunderthehood-WALstructure So infinite storage of archive is definitely not necessary for vanilla open-source version, but archive itself is needed. Sincerely, Dmitriy Pavlov ср, 11 нояб. 2020 г. в 01:21, Raymond Wilson : > Isn't the discussion here related to the WAL archive? If you disable that > don't you still have the WAL containing un-checkpointed changes? > > On Wed, Nov 11, 2020 at 11:01 AM Dmitriy Pavlov > wrote: > > > Hi Denis, > > > > the short answer here, Apache Ignite guarantees ACID, and for > D-Durability > > it is required to save all changes in some WAL/Redo Log to have a safe > way > > to recover from any hardware failures/disk outage. > > > > Should the user disable WAL, he/she could potentially lose durability. > > > > Sincerely, > > Dmitriy Pavlov > > > > вт, 10 нояб. 2020 г. в 09:57, ткаленко кирилл : > > > > > Hello guys again! > > > > > > Does anyone know why we are doing any calculation here > > > IgniteUtils#adjustedWalHistorySize at all? > > > Would it be easier to always take the > > > DataStorageConfiguration#maxWalArchiveSize? It seems that the user can > > > easily do this himself by changing the value by 1 byte. > > > > > > 06.11.2020, 13:56, "Ivan Daschinsky" : > > > > Alex, thanks for pointing that out. Shame that I missed it. > > > > > > > > пт, 6 нояб. 2020 г. в 13:45, Alex Plehanov >: > > > > > > > >> Guys, > > > >> > > > >> We already have > > FileWriteAheadLogManager#maxSegCountWithoutCheckpoint. > > > >> Checkpoint triggered if there are too many WAL segments without > > > checkpoint. > > > >> Looks like you are talking about this feature. > > > >> > > > >> пт, 6 нояб. 2020 г. в 13:21, Ivan Daschinsky >: > > > >> > > > >> > Kirill and I discussed privately proposed approach. As far as I > > > >> understand, > > > >> > Kirill suggests to implement some > > > >> > heuristic to do a force checkpoint in some cases if user by > mistake > > > >> > misconfigured cluster in order to preserve > > > >> > requested size of WAL archive. > > > >> > Currently, as for me, this approach is questionable, because it > can > > > cause > > > >> > some performance problems. But as an option, > > > >> > it can be used and should be switchable. > > > >> > > > > >> > пт, 6 нояб. 2020 г. в 12:36, Ivan Daschinsky < > ivanda...@gmail.com > > >: > > > >> > > > > >> > > Kirill, how your approach will help if user tuned a cluster to > do > > > >> > > checkpoints rarely under load? > > > >> > > No way. > > > >> > > > > > >> > > пт, 6 нояб. 2020 г. в 12:19, ткаленко кирилл < > > tkalkir...@yandex.ru > > > >: > > > >> > > > > > >> > >> Ivan, I agree with you that the archive is primarily about > > > >> optimization. > > > >> > >> > > > >> > >> If the size of the archive is critical for the user, we have > no > > > >> > >> protection against this, we can always go beyond this limit. > > > >> > >> Thus, the user needs to remember this and configure it in some > > > way. > > > >> > >> > > > >> > >> I suggest not to exceed this limit and give the expected > > behavior > > > for > > > >> > the > > > >> > >> user. At the same time, the segments needed for recovery will > > > remain > > > >> and > > > >> > >> there will be no data loss. > > > >> > >> > > > >> > >> 06.11.2020, 11:29, "Ivan Daschinsky" : > > > >> > >> > Guys, fisrt of all, archiving is not for PITR at all, this > is > > > >> > >> optimization. > > > >> > >> > If we disable archiving, every rollover we need to create > new > > > file. > > > >> If > > > >> > >> we > > > >> > >> > enable archiving, we reserve 10 (by default) segments filled > > > with > > > >> > >> zeroes. > > > >> > >> > We use mmap by default, so if we use no-archiver approach: > > > >> > >> > 1. We firstly create new empty file > > > >> > >> > 2. Call on it sun.nio.ch.FileChannelImpl#map, thats under > the > > > hood > > > >> > >> > a. If file is shorter, than wal segment size, it > > > >> > >> > calls sun.nio.ch.FileDispatcherImpl#truncate0, this is under > > the > > > >> hood > > > >> > >> just > > > >> > >> > a system call truncate [1] > > > >> > >> > b. Than it calls system call mmap on this > > > >> > >> > file sun.nio.ch.FileChannelImpl#map0, under the hood see [2] > > > >> > >> > These manipulation are not free and cheap. So rollover will > be > > > much > > > >> > much > > > >> > >> > slower. > > > >> > >> > If archiving is enabled, 10 segments are already > preallocated > > >
Re: Ignite 2.9 Announcement Blog Post
ok, I see, I supposed this list is somehow automatically being built or related to announcements. ср, 11 нояб. 2020 г. в 01:48, Denis Magda : > You also need to add a release to the reporter manually just for tracking > purposes. I used to do this once a quarter when the board was requesting a > report and I would add all the missing releases into that system while > preparing the report. > > - > Denis > > > On Tue, Nov 10, 2020 at 1:27 PM Dmitriy Pavlov wrote: > > > Hi Alex, > > > > Thank you for driving this release and this post. > > > > Was the release announced at announce@ ? > > > > Denis, Ignite veterans, > > > > The reason I'm asking: this release was not available at > > https://reporter.apache.org/addrelease.html?ignite ? I've added it now. > Is > > it always required to add it there manually? > > > > Sincerely, > > Dmitriy Pavlov > > > > пт, 6 нояб. 2020 г. в 09:38, Alex Plehanov : > > > > > Igniters, > > > > > > Ignite 2.9 announce blog post is published now: [1] > > > > > > [1] : https://blogs.apache.org/ignite/entry/apache-ignite-2-9-released > > > > > > пт, 25 сент. 2020 г. в 11:04, Alex Plehanov : > > > > > > > Denis, > > > > > > > > Ok, I will prepare the text soon. > > > > > > > > чт, 24 сент. 2020 г. в 23:36, Denis Magda : > > > > > > > >> @Alex Plehanov > > > >> > > > >> Are you interested in writing a blog post that covers the major > > > >> improvements of the 2.9 release? It will be referenced from the > > > >> announcement email and distributed through other channels. Examples > of > > > >> previous articles: https://blogs.apache.org/ignite/ > > > >> > > > >> It happened that I was the only one who would write such an article > > but > > > >> will step down with pleasure if you can take over this task. Once > the > > > >> draft > > > >> is ready, I'll ask a professional technical editor, who reviews my > > > texts, > > > >> to work with you as well. > > > >> > > > >> - > > > >> Denis > > > >> > > > > > > > > > >
Re: Ignite 2.9 Announcement Blog Post
You also need to add a release to the reporter manually just for tracking purposes. I used to do this once a quarter when the board was requesting a report and I would add all the missing releases into that system while preparing the report. - Denis On Tue, Nov 10, 2020 at 1:27 PM Dmitriy Pavlov wrote: > Hi Alex, > > Thank you for driving this release and this post. > > Was the release announced at announce@ ? > > Denis, Ignite veterans, > > The reason I'm asking: this release was not available at > https://reporter.apache.org/addrelease.html?ignite ? I've added it now. Is > it always required to add it there manually? > > Sincerely, > Dmitriy Pavlov > > пт, 6 нояб. 2020 г. в 09:38, Alex Plehanov : > > > Igniters, > > > > Ignite 2.9 announce blog post is published now: [1] > > > > [1] : https://blogs.apache.org/ignite/entry/apache-ignite-2-9-released > > > > пт, 25 сент. 2020 г. в 11:04, Alex Plehanov : > > > > > Denis, > > > > > > Ok, I will prepare the text soon. > > > > > > чт, 24 сент. 2020 г. в 23:36, Denis Magda : > > > > > >> @Alex Plehanov > > >> > > >> Are you interested in writing a blog post that covers the major > > >> improvements of the 2.9 release? It will be referenced from the > > >> announcement email and distributed through other channels. Examples of > > >> previous articles: https://blogs.apache.org/ignite/ > > >> > > >> It happened that I was the only one who would write such an article > but > > >> will step down with pleasure if you can take over this task. Once the > > >> draft > > >> is ready, I'll ask a professional technical editor, who reviews my > > texts, > > >> to work with you as well. > > >> > > >> - > > >> Denis > > >> > > > > > >
Re: Apache Ignite Board Report, Nov 11
Looks good to me. Thanks for sharing the draft. - Denis On Tue, Nov 10, 2020 at 1:55 PM Dmitriy Pavlov wrote: > Hi Igniters, > > The report for the Board meeting on Nov 18 is due today. > > I've appended the current version of the report at the end of the email. > The same report has been filed to the agenda. > > If you feel that we can add some crucial points to the report, please let > me know. I guess I still can do some updates. > > Sincerely, > Dmitriy Pavlov > > ## Description: > The mission of Ignite is the creation and maintenance of software related > to > High-performance, integrated and distributed In-Memory Database and Caching > Platform providing in-memory data caching, partitioning, processing, and > querying components. > > ## Issues: > There are no issues requiring board attention. > > ## Membership Data: > Apache Ignite was founded 2015-08-19 (5 years ago) > There are currently 54 committers and 34 PMC members in this project. > The Committer-to-PMC ratio is roughly 7:5. > > Community changes, past quarter: > - No new PMC members. The last addition was Maxim Muzafarov on 2020-04-28. > - Sergey Stronchinskiy was added as a committer on 2020-10-14 > > ## Project Activity: > - Major release 2.9.0 was completed on October > (https://blogs.apache.org/ignite/entry/apache-ignite-2-9-released) > - Minor release 2.9.1 and next major release 2.10.0 are under discussion > - Community members decided to introduce Apache Ignite as a database (no > trademark changes are expected, but the project description needs an > update). During entering to the Incubator Ignite was In-memory data grid > - Community members discuss approaches on how to develop 3.0.0. There are > two > options - separate repository with development of some modules from > scratch > preserving only behavior. The alternative is to develop in evolutionary > mode > using the existing code base changing it several times. > > ## Community Health: > - dev@ mailing list and amount of GitHub-related notifications are almost > the > same. > - user@ mailing list and amount of PRs/JIRA tickets has a slight decrease > -10...-40% > - Community members continue to write posts and run talks at online > meetups >
Re: Why WAL archives enabled by default?
Isn't the discussion here related to the WAL archive? If you disable that don't you still have the WAL containing un-checkpointed changes? On Wed, Nov 11, 2020 at 11:01 AM Dmitriy Pavlov wrote: > Hi Denis, > > the short answer here, Apache Ignite guarantees ACID, and for D-Durability > it is required to save all changes in some WAL/Redo Log to have a safe way > to recover from any hardware failures/disk outage. > > Should the user disable WAL, he/she could potentially lose durability. > > Sincerely, > Dmitriy Pavlov > > вт, 10 нояб. 2020 г. в 09:57, ткаленко кирилл : > > > Hello guys again! > > > > Does anyone know why we are doing any calculation here > > IgniteUtils#adjustedWalHistorySize at all? > > Would it be easier to always take the > > DataStorageConfiguration#maxWalArchiveSize? It seems that the user can > > easily do this himself by changing the value by 1 byte. > > > > 06.11.2020, 13:56, "Ivan Daschinsky" : > > > Alex, thanks for pointing that out. Shame that I missed it. > > > > > > пт, 6 нояб. 2020 г. в 13:45, Alex Plehanov : > > > > > >> Guys, > > >> > > >> We already have > FileWriteAheadLogManager#maxSegCountWithoutCheckpoint. > > >> Checkpoint triggered if there are too many WAL segments without > > checkpoint. > > >> Looks like you are talking about this feature. > > >> > > >> пт, 6 нояб. 2020 г. в 13:21, Ivan Daschinsky : > > >> > > >> > Kirill and I discussed privately proposed approach. As far as I > > >> understand, > > >> > Kirill suggests to implement some > > >> > heuristic to do a force checkpoint in some cases if user by mistake > > >> > misconfigured cluster in order to preserve > > >> > requested size of WAL archive. > > >> > Currently, as for me, this approach is questionable, because it can > > cause > > >> > some performance problems. But as an option, > > >> > it can be used and should be switchable. > > >> > > > >> > пт, 6 нояб. 2020 г. в 12:36, Ivan Daschinsky >: > > >> > > > >> > > Kirill, how your approach will help if user tuned a cluster to do > > >> > > checkpoints rarely under load? > > >> > > No way. > > >> > > > > >> > > пт, 6 нояб. 2020 г. в 12:19, ткаленко кирилл < > tkalkir...@yandex.ru > > >: > > >> > > > > >> > >> Ivan, I agree with you that the archive is primarily about > > >> optimization. > > >> > >> > > >> > >> If the size of the archive is critical for the user, we have no > > >> > >> protection against this, we can always go beyond this limit. > > >> > >> Thus, the user needs to remember this and configure it in some > > way. > > >> > >> > > >> > >> I suggest not to exceed this limit and give the expected > behavior > > for > > >> > the > > >> > >> user. At the same time, the segments needed for recovery will > > remain > > >> and > > >> > >> there will be no data loss. > > >> > >> > > >> > >> 06.11.2020, 11:29, "Ivan Daschinsky" : > > >> > >> > Guys, fisrt of all, archiving is not for PITR at all, this is > > >> > >> optimization. > > >> > >> > If we disable archiving, every rollover we need to create new > > file. > > >> If > > >> > >> we > > >> > >> > enable archiving, we reserve 10 (by default) segments filled > > with > > >> > >> zeroes. > > >> > >> > We use mmap by default, so if we use no-archiver approach: > > >> > >> > 1. We firstly create new empty file > > >> > >> > 2. Call on it sun.nio.ch.FileChannelImpl#map, thats under the > > hood > > >> > >> > a. If file is shorter, than wal segment size, it > > >> > >> > calls sun.nio.ch.FileDispatcherImpl#truncate0, this is under > the > > >> hood > > >> > >> just > > >> > >> > a system call truncate [1] > > >> > >> > b. Than it calls system call mmap on this > > >> > >> > file sun.nio.ch.FileChannelImpl#map0, under the hood see [2] > > >> > >> > These manipulation are not free and cheap. So rollover will be > > much > > >> > much > > >> > >> > slower. > > >> > >> > If archiving is enabled, 10 segments are already preallocated > > at the > > >> > >> moment > > >> > >> > of node's start. > > >> > >> > > > >> > >> > When archiving is enabled, archiver just copy previous > > preallocated > > >> > >> segment > > >> > >> > and move it to archive directory. > > >> > >> > This archived segment is crucial for recovery. When new > > checkpoints > > >> > >> > finished, all eligible for trunocating segments are just > > removed. > > >> > >> > > > >> > >> > If archiving is disabled, we also write WAL segments in wal > > >> directory > > >> > >> and > > >> > >> > disabling archiving don't prevent you from storing segments, > if > > they > > >> > are > > >> > >> > required for recovery. > > >> > >> > > > >> > >> >>> Before increasing the size of WAL archive (transferring to > > archive > > >> > >> > > > >> > >> > /rollOver, compression, decompression), we can make sure that > > there > > >> > >> will be > > >> > >> > enough space in the archive and if there is no such, then we > > will > > >> try > > >> > to > > >> > >> >>> clean
Re: Why WAL archives enabled by default?
Hi Denis, the short answer here, Apache Ignite guarantees ACID, and for D-Durability it is required to save all changes in some WAL/Redo Log to have a safe way to recover from any hardware failures/disk outage. Should the user disable WAL, he/she could potentially lose durability. Sincerely, Dmitriy Pavlov вт, 10 нояб. 2020 г. в 09:57, ткаленко кирилл : > Hello guys again! > > Does anyone know why we are doing any calculation here > IgniteUtils#adjustedWalHistorySize at all? > Would it be easier to always take the > DataStorageConfiguration#maxWalArchiveSize? It seems that the user can > easily do this himself by changing the value by 1 byte. > > 06.11.2020, 13:56, "Ivan Daschinsky" : > > Alex, thanks for pointing that out. Shame that I missed it. > > > > пт, 6 нояб. 2020 г. в 13:45, Alex Plehanov : > > > >> Guys, > >> > >> We already have FileWriteAheadLogManager#maxSegCountWithoutCheckpoint. > >> Checkpoint triggered if there are too many WAL segments without > checkpoint. > >> Looks like you are talking about this feature. > >> > >> пт, 6 нояб. 2020 г. в 13:21, Ivan Daschinsky : > >> > >> > Kirill and I discussed privately proposed approach. As far as I > >> understand, > >> > Kirill suggests to implement some > >> > heuristic to do a force checkpoint in some cases if user by mistake > >> > misconfigured cluster in order to preserve > >> > requested size of WAL archive. > >> > Currently, as for me, this approach is questionable, because it can > cause > >> > some performance problems. But as an option, > >> > it can be used and should be switchable. > >> > > >> > пт, 6 нояб. 2020 г. в 12:36, Ivan Daschinsky : > >> > > >> > > Kirill, how your approach will help if user tuned a cluster to do > >> > > checkpoints rarely under load? > >> > > No way. > >> > > > >> > > пт, 6 нояб. 2020 г. в 12:19, ткаленко кирилл >: > >> > > > >> > >> Ivan, I agree with you that the archive is primarily about > >> optimization. > >> > >> > >> > >> If the size of the archive is critical for the user, we have no > >> > >> protection against this, we can always go beyond this limit. > >> > >> Thus, the user needs to remember this and configure it in some > way. > >> > >> > >> > >> I suggest not to exceed this limit and give the expected behavior > for > >> > the > >> > >> user. At the same time, the segments needed for recovery will > remain > >> and > >> > >> there will be no data loss. > >> > >> > >> > >> 06.11.2020, 11:29, "Ivan Daschinsky" : > >> > >> > Guys, fisrt of all, archiving is not for PITR at all, this is > >> > >> optimization. > >> > >> > If we disable archiving, every rollover we need to create new > file. > >> If > >> > >> we > >> > >> > enable archiving, we reserve 10 (by default) segments filled > with > >> > >> zeroes. > >> > >> > We use mmap by default, so if we use no-archiver approach: > >> > >> > 1. We firstly create new empty file > >> > >> > 2. Call on it sun.nio.ch.FileChannelImpl#map, thats under the > hood > >> > >> > a. If file is shorter, than wal segment size, it > >> > >> > calls sun.nio.ch.FileDispatcherImpl#truncate0, this is under the > >> hood > >> > >> just > >> > >> > a system call truncate [1] > >> > >> > b. Than it calls system call mmap on this > >> > >> > file sun.nio.ch.FileChannelImpl#map0, under the hood see [2] > >> > >> > These manipulation are not free and cheap. So rollover will be > much > >> > much > >> > >> > slower. > >> > >> > If archiving is enabled, 10 segments are already preallocated > at the > >> > >> moment > >> > >> > of node's start. > >> > >> > > >> > >> > When archiving is enabled, archiver just copy previous > preallocated > >> > >> segment > >> > >> > and move it to archive directory. > >> > >> > This archived segment is crucial for recovery. When new > checkpoints > >> > >> > finished, all eligible for trunocating segments are just > removed. > >> > >> > > >> > >> > If archiving is disabled, we also write WAL segments in wal > >> directory > >> > >> and > >> > >> > disabling archiving don't prevent you from storing segments, if > they > >> > are > >> > >> > required for recovery. > >> > >> > > >> > >> >>> Before increasing the size of WAL archive (transferring to > archive > >> > >> > > >> > >> > /rollOver, compression, decompression), we can make sure that > there > >> > >> will be > >> > >> > enough space in the archive and if there is no such, then we > will > >> try > >> > to > >> > >> >>> clean it. We cannot delete those segments that are required > for > >> > >> recovery > >> > >> > > >> > >> > (between the last two checkpoints) and reserved for example for > >> > >> historical > >> > >> > rebalancing. > >> > >> > First of all, compression/decompression is offtopic here. > >> > >> > Secondly, wal segments are required only with idx higher than > LAST > >> > >> > checkpoint marker. > >> > >> > Thirdly, archiving and rolling over can be during checkpoint > and we
Re: Ignite 2.9 Announcement Blog Post
Ok, thanks for sharing the link. ср, 11 нояб. 2020 г. в 00:35, Alex Plehanov : > Dmitriy, > > Yes, the release was announced at announce@. See [1]. > > [1] : > > https://lists.apache.org/thread.html/r2e3dbf7a055fdbc57f47c1180afea59dd8a335c8dde7a5e53885e861%40%3Cannounce.apache.org%3E > > > ср, 11 нояб. 2020 г. в 00:27, Dmitriy Pavlov : > > > Hi Alex, > > > > Thank you for driving this release and this post. > > > > Was the release announced at announce@ ? > > > > Denis, Ignite veterans, > > > > The reason I'm asking: this release was not available at > > https://reporter.apache.org/addrelease.html?ignite ? I've added it now. > Is > > it always required to add it there manually? > > > > Sincerely, > > Dmitriy Pavlov > > > > пт, 6 нояб. 2020 г. в 09:38, Alex Plehanov : > > > > > Igniters, > > > > > > Ignite 2.9 announce blog post is published now: [1] > > > > > > [1] : https://blogs.apache.org/ignite/entry/apache-ignite-2-9-released > > > > > > пт, 25 сент. 2020 г. в 11:04, Alex Plehanov : > > > > > > > Denis, > > > > > > > > Ok, I will prepare the text soon. > > > > > > > > чт, 24 сент. 2020 г. в 23:36, Denis Magda : > > > > > > > >> @Alex Plehanov > > > >> > > > >> Are you interested in writing a blog post that covers the major > > > >> improvements of the 2.9 release? It will be referenced from the > > > >> announcement email and distributed through other channels. Examples > of > > > >> previous articles: https://blogs.apache.org/ignite/ > > > >> > > > >> It happened that I was the only one who would write such an article > > but > > > >> will step down with pleasure if you can take over this task. Once > the > > > >> draft > > > >> is ready, I'll ask a professional technical editor, who reviews my > > > texts, > > > >> to work with you as well. > > > >> > > > >> - > > > >> Denis > > > >> > > > > > > > > > >
Apache Ignite Board Report, Nov 11
Hi Igniters, The report for the Board meeting on Nov 18 is due today. I've appended the current version of the report at the end of the email. The same report has been filed to the agenda. If you feel that we can add some crucial points to the report, please let me know. I guess I still can do some updates. Sincerely, Dmitriy Pavlov ## Description: The mission of Ignite is the creation and maintenance of software related to High-performance, integrated and distributed In-Memory Database and Caching Platform providing in-memory data caching, partitioning, processing, and querying components. ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Ignite was founded 2015-08-19 (5 years ago) There are currently 54 committers and 34 PMC members in this project. The Committer-to-PMC ratio is roughly 7:5. Community changes, past quarter: - No new PMC members. The last addition was Maxim Muzafarov on 2020-04-28. - Sergey Stronchinskiy was added as a committer on 2020-10-14 ## Project Activity: - Major release 2.9.0 was completed on October (https://blogs.apache.org/ignite/entry/apache-ignite-2-9-released) - Minor release 2.9.1 and next major release 2.10.0 are under discussion - Community members decided to introduce Apache Ignite as a database (no trademark changes are expected, but the project description needs an update). During entering to the Incubator Ignite was In-memory data grid - Community members discuss approaches on how to develop 3.0.0. There are two options - separate repository with development of some modules from scratch preserving only behavior. The alternative is to develop in evolutionary mode using the existing code base changing it several times. ## Community Health: - dev@ mailing list and amount of GitHub-related notifications are almost the same. - user@ mailing list and amount of PRs/JIRA tickets has a slight decrease -10...-40% - Community members continue to write posts and run talks at online meetups
Re: Ignite 2.9 Announcement Blog Post
Dmitriy, Yes, the release was announced at announce@. See [1]. [1] : https://lists.apache.org/thread.html/r2e3dbf7a055fdbc57f47c1180afea59dd8a335c8dde7a5e53885e861%40%3Cannounce.apache.org%3E ср, 11 нояб. 2020 г. в 00:27, Dmitriy Pavlov : > Hi Alex, > > Thank you for driving this release and this post. > > Was the release announced at announce@ ? > > Denis, Ignite veterans, > > The reason I'm asking: this release was not available at > https://reporter.apache.org/addrelease.html?ignite ? I've added it now. Is > it always required to add it there manually? > > Sincerely, > Dmitriy Pavlov > > пт, 6 нояб. 2020 г. в 09:38, Alex Plehanov : > > > Igniters, > > > > Ignite 2.9 announce blog post is published now: [1] > > > > [1] : https://blogs.apache.org/ignite/entry/apache-ignite-2-9-released > > > > пт, 25 сент. 2020 г. в 11:04, Alex Plehanov : > > > > > Denis, > > > > > > Ok, I will prepare the text soon. > > > > > > чт, 24 сент. 2020 г. в 23:36, Denis Magda : > > > > > >> @Alex Plehanov > > >> > > >> Are you interested in writing a blog post that covers the major > > >> improvements of the 2.9 release? It will be referenced from the > > >> announcement email and distributed through other channels. Examples of > > >> previous articles: https://blogs.apache.org/ignite/ > > >> > > >> It happened that I was the only one who would write such an article > but > > >> will step down with pleasure if you can take over this task. Once the > > >> draft > > >> is ready, I'll ask a professional technical editor, who reviews my > > texts, > > >> to work with you as well. > > >> > > >> - > > >> Denis > > >> > > > > > >
Re: Ignite 2.9 Announcement Blog Post
Hi Alex, Thank you for driving this release and this post. Was the release announced at announce@ ? Denis, Ignite veterans, The reason I'm asking: this release was not available at https://reporter.apache.org/addrelease.html?ignite ? I've added it now. Is it always required to add it there manually? Sincerely, Dmitriy Pavlov пт, 6 нояб. 2020 г. в 09:38, Alex Plehanov : > Igniters, > > Ignite 2.9 announce blog post is published now: [1] > > [1] : https://blogs.apache.org/ignite/entry/apache-ignite-2-9-released > > пт, 25 сент. 2020 г. в 11:04, Alex Plehanov : > > > Denis, > > > > Ok, I will prepare the text soon. > > > > чт, 24 сент. 2020 г. в 23:36, Denis Magda : > > > >> @Alex Plehanov > >> > >> Are you interested in writing a blog post that covers the major > >> improvements of the 2.9 release? It will be referenced from the > >> announcement email and distributed through other channels. Examples of > >> previous articles: https://blogs.apache.org/ignite/ > >> > >> It happened that I was the only one who would write such an article but > >> will step down with pleasure if you can take over this task. Once the > >> draft > >> is ready, I'll ask a professional technical editor, who reviews my > texts, > >> to work with you as well. > >> > >> - > >> Denis > >> > > >
Re: [DISCUSS] Ignite 3.0 development approach
Sergey, Your summary makes sense to me. However, how we come up from *Development transparency* to *create a separate public repository dedicated for 3.0*? For me *development transparency* is about making changes in the master branch. These changes will definitely be seen by all the Ignite developers. A dedicated public repository is technically public and visible for everyone, but it allows development without IEPs, without public discussion (since all the code changes are not related to the master branch) it also allows a large number of assumptions and deviations (like code-style violations). It also not about *development transparency* since developers which are working on 3.0 is only a subset of all Ignite developers which may continue working on 2.x. For me, this would be a huge step backwards. Ignite veterans should remember how long the branch stabilization took for the 2.x version with the PDS. I think each breaking change should be passed through the master branch. On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov wrote: > > Makes sense to me. > > вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov : > > > Igniters, > > > > I thought over Friday meeting ideas and concerns and summarized them in > > these three points: > > > > > >1. *Components design unification approach.* New proposed components > >will be developed by different contributors, but they need to be unified > >and should integrate with each other easily. To ensure that I suggest > >calling an architecture group that will create design guidelines for all > >components and high-level overview of overall architecture. How code is > >split into components, what are component boundaries, how component > >lifecycle works and what are its interfaces - all these and other > > questions > >should be covered. > > > >2. *Scope management.* Apache 3.0 should be implemented within a > >reasonable time, so we need some procedure to decide whether a > > particular > >feature should be dropped from the scope of 3.0 and postponed to 3.1 > >release. To do so I suggest to range all features by two parameters: > >criticality for 3.0 and amount of breaking changes. 3.0 scope should > >include features of high criticality AND features with a big amount of > >breaking changes. All other features can be made optional. > > > >3. *Development transparency.* Development of all components should be > >made as transparent for everyone as possible. Any contributor should be > >able to look over any component at any stage of development. To achieve > >this I suggest to create a separate public repository dedicated for 3.0 > >development. It will make the code available for everyone but when > >development of 3.0 is done we won't loose any stars of our current > >repository as we merge dev repo into main one and drop dev. > > > > Do these ideas make sense to you? Are there any concerns not covered by > > these suggestions? > > > > On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova > > > > wrote: > > > > > Here are the slides from Alexey Goncharuk. Let's think this over and > > > continue on Monday: > > > > > > > > https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf > > > > > > чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov : > > > > > > > Folks, > > > > > > > > Should we perform cleanup work before (r)evolutional changes? > > > > My huge proposal is to get rid of things which we don't need anyway > > > > - local caches, > > > > - strange tx modes, > > > > - code overcomplexity because of RollingUpgrade feature never attended > > at > > > > AI, > > > > - etc, > > > > before choosing the way. > > > > > > > > On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko < > > > > valentin.kuliche...@gmail.com> wrote: > > > > > > > > > Ksenia, thanks for scheduling this on such short notice! > > > > > > > > > > As for the original topic, I do support Alexey's idea. We're not > > going > > > to > > > > > rewrite anything from scratch, as most of the components are going to > > > be > > > > > moved as-is or with minimal modifications. However, the changes that > > > are > > > > > proposed imply serious rework of the core parts of the code, which > > are > > > > not > > > > > properly decoupled from each other and from other parts. This makes > > the > > > > > incremental approach borderline impossible. Developing in a new repo, > > > > > however, addresses this concern. As a bonus, we can also refactor the > > > > code, > > > > > introduce better decoupling, get rid of kernel context, and develop > > > unit > > > > > tests (finally!). > > > > > > > > > > Basically, this proposal only affects the *process*, not the set of > > > > changes > > > > > we had discussed before. Ignite 3.0 is our unique chance to make > > things > > > > > right. > > > > > > > > > > -Val > > > > > > > > > > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova < > > > > romanova.ks@gma
Re: [DISCUSS] Ignite 3.0 development approach
Makes sense to me. вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov : > Igniters, > > I thought over Friday meeting ideas and concerns and summarized them in > these three points: > > >1. *Components design unification approach.* New proposed components >will be developed by different contributors, but they need to be unified >and should integrate with each other easily. To ensure that I suggest >calling an architecture group that will create design guidelines for all >components and high-level overview of overall architecture. How code is >split into components, what are component boundaries, how component >lifecycle works and what are its interfaces - all these and other > questions >should be covered. > >2. *Scope management.* Apache 3.0 should be implemented within a >reasonable time, so we need some procedure to decide whether a > particular >feature should be dropped from the scope of 3.0 and postponed to 3.1 >release. To do so I suggest to range all features by two parameters: >criticality for 3.0 and amount of breaking changes. 3.0 scope should >include features of high criticality AND features with a big amount of >breaking changes. All other features can be made optional. > >3. *Development transparency.* Development of all components should be >made as transparent for everyone as possible. Any contributor should be >able to look over any component at any stage of development. To achieve >this I suggest to create a separate public repository dedicated for 3.0 >development. It will make the code available for everyone but when >development of 3.0 is done we won't loose any stars of our current >repository as we merge dev repo into main one and drop dev. > > Do these ideas make sense to you? Are there any concerns not covered by > these suggestions? > > On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova > > wrote: > > > Here are the slides from Alexey Goncharuk. Let's think this over and > > continue on Monday: > > > > > https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf > > > > чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov : > > > > > Folks, > > > > > > Should we perform cleanup work before (r)evolutional changes? > > > My huge proposal is to get rid of things which we don't need anyway > > > - local caches, > > > - strange tx modes, > > > - code overcomplexity because of RollingUpgrade feature never attended > at > > > AI, > > > - etc, > > > before choosing the way. > > > > > > On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko < > > > valentin.kuliche...@gmail.com> wrote: > > > > > > > Ksenia, thanks for scheduling this on such short notice! > > > > > > > > As for the original topic, I do support Alexey's idea. We're not > going > > to > > > > rewrite anything from scratch, as most of the components are going to > > be > > > > moved as-is or with minimal modifications. However, the changes that > > are > > > > proposed imply serious rework of the core parts of the code, which > are > > > not > > > > properly decoupled from each other and from other parts. This makes > the > > > > incremental approach borderline impossible. Developing in a new repo, > > > > however, addresses this concern. As a bonus, we can also refactor the > > > code, > > > > introduce better decoupling, get rid of kernel context, and develop > > unit > > > > tests (finally!). > > > > > > > > Basically, this proposal only affects the *process*, not the set of > > > changes > > > > we had discussed before. Ignite 3.0 is our unique chance to make > things > > > > right. > > > > > > > > -Val > > > > > > > > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova < > > > romanova.ks@gmail.com > > > > > > > > > wrote: > > > > > > > > > Pavel, all the interesting points will be anyway published here in > > > > English > > > > > (as the principal "if it's not on devlist it doesn't happened" is > > still > > > > > relevant). This is just a quick call for a group of developers. > Later > > > we > > > > > can do a separate presentation of idea and discussion in English as > > we > > > > did > > > > > for the Ignite 3.0 draft of changes. > > > > > > > > > > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn >: > > > > > > > > > > > Kseniya, > > > > > > > > > > > > Thanks for scheduling this call. > > > > > > Do you think we can switch to English if non-Russian speaking > > > community > > > > > > members decide to join? > > > > > > > > > > > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova < > > > > > romanova.ks@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Let's do this community discussion open. Here's the link on > zoom > > > call > > > > > in > > > > > > > Russian for Friday 6 PM: > > > > > > > > > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/ > > > > > > > > > > > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov < > > nizhi...@apache.org > > > >: > > > > > > > > > >
[jira] [Created] (IGNITE-13692) .NET: Default query timeout
Pavel Tupitsyn created IGNITE-13692:
---
Summary: .NET: Default query timeout
Key: IGNITE-13692
URL: https://issues.apache.org/jira/browse/IGNITE-13692
Project: Ignite
Issue Type: Improvement
Components: platforms
Affects Versions: 2.9
Reporter: Pavel Tupitsyn
Fix For: 2.10
Propagate IGNITE-7285 changes to .NET thin and thick APIs:
* Add {{IgniteConfiguration.DefaultQueryTimeout}}
* Change {{SqlFieldsQuery.Timeout}} to be {{-1}} by default, so that global
config is used
* Enable {{ClientBitmaskFeature.DefaultQueryTimeout}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Ignite 3.0 development approach
Igniters, I thought over Friday meeting ideas and concerns and summarized them in these three points: 1. *Components design unification approach.* New proposed components will be developed by different contributors, but they need to be unified and should integrate with each other easily. To ensure that I suggest calling an architecture group that will create design guidelines for all components and high-level overview of overall architecture. How code is split into components, what are component boundaries, how component lifecycle works and what are its interfaces - all these and other questions should be covered. 2. *Scope management.* Apache 3.0 should be implemented within a reasonable time, so we need some procedure to decide whether a particular feature should be dropped from the scope of 3.0 and postponed to 3.1 release. To do so I suggest to range all features by two parameters: criticality for 3.0 and amount of breaking changes. 3.0 scope should include features of high criticality AND features with a big amount of breaking changes. All other features can be made optional. 3. *Development transparency.* Development of all components should be made as transparent for everyone as possible. Any contributor should be able to look over any component at any stage of development. To achieve this I suggest to create a separate public repository dedicated for 3.0 development. It will make the code available for everyone but when development of 3.0 is done we won't loose any stars of our current repository as we merge dev repo into main one and drop dev. Do these ideas make sense to you? Are there any concerns not covered by these suggestions? On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova wrote: > Here are the slides from Alexey Goncharuk. Let's think this over and > continue on Monday: > > https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf > > чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov : > > > Folks, > > > > Should we perform cleanup work before (r)evolutional changes? > > My huge proposal is to get rid of things which we don't need anyway > > - local caches, > > - strange tx modes, > > - code overcomplexity because of RollingUpgrade feature never attended at > > AI, > > - etc, > > before choosing the way. > > > > On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko < > > valentin.kuliche...@gmail.com> wrote: > > > > > Ksenia, thanks for scheduling this on such short notice! > > > > > > As for the original topic, I do support Alexey's idea. We're not going > to > > > rewrite anything from scratch, as most of the components are going to > be > > > moved as-is or with minimal modifications. However, the changes that > are > > > proposed imply serious rework of the core parts of the code, which are > > not > > > properly decoupled from each other and from other parts. This makes the > > > incremental approach borderline impossible. Developing in a new repo, > > > however, addresses this concern. As a bonus, we can also refactor the > > code, > > > introduce better decoupling, get rid of kernel context, and develop > unit > > > tests (finally!). > > > > > > Basically, this proposal only affects the *process*, not the set of > > changes > > > we had discussed before. Ignite 3.0 is our unique chance to make things > > > right. > > > > > > -Val > > > > > > On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova < > > romanova.ks@gmail.com > > > > > > > wrote: > > > > > > > Pavel, all the interesting points will be anyway published here in > > > English > > > > (as the principal "if it's not on devlist it doesn't happened" is > still > > > > relevant). This is just a quick call for a group of developers. Later > > we > > > > can do a separate presentation of idea and discussion in English as > we > > > did > > > > for the Ignite 3.0 draft of changes. > > > > > > > > вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn : > > > > > > > > > Kseniya, > > > > > > > > > > Thanks for scheduling this call. > > > > > Do you think we can switch to English if non-Russian speaking > > community > > > > > members decide to join? > > > > > > > > > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova < > > > > romanova.ks@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > Let's do this community discussion open. Here's the link on zoom > > call > > > > in > > > > > > Russian for Friday 6 PM: > > > > > > > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/ > > > > > > > > > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov < > nizhi...@apache.org > > >: > > > > > > > > > > > > > Time works for me. > > > > > > > > > > > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk < > > > > > alexey.goncha...@gmail.com > > > > > > > > > > > > > > написал(а): > > > > > > > > > > > > > > > > Nikolay, > > > > > > > > > > > > > > > > I am up for the call. I will try to explain my reasoning in > > > greater > > > >
[jira] [Created] (IGNITE-13691) Calcite Integration. Support of EXCEPT operator
Yury Gerzhedovich created IGNITE-13691: -- Summary: Calcite Integration. Support of EXCEPT operator Key: IGNITE-13691 URL: https://issues.apache.org/jira/browse/IGNITE-13691 Project: Ignite Issue Type: Improvement Components: sql Reporter: Yury Gerzhedovich Assignee: Taras Ledkov As of now new Calcite based SQL engine doesn't support opearator EXCEPT. We need to implement it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSSION] Apache Ignite Release 2.10 (time, scope, manager)
Folks, I've created the release page on the Confluence. Feel free to mail in case of any errors. https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.10 On Fri, 6 Nov 2020 at 17:56, Maxim Muzafarov wrote: > > Igniters, > > > Let's finalize the discussion [1] about the next upcoming major Apache > Ignite 2.10 release. The major improvements related to the proposed > release: > - Improvements for partition clearing related parts > - Add tracing of SQL queries. > - CPP: Implement Cluster API > - .NET: Thin client: Transactions > - .NET: Thin Client: Continuous Query > - Java Thin client Kubernetes discovery > > etc. > Total: 166 RESOLVED issues [2]. > > > Let's start the discussion about Time and Scope, and also I propose > myself as the release manager of the Apache Ignite 2.10. If you'd like > to lead this release, please, let us know, I see no problems to chose > a better candidate. > > > Proposed release timeline: > > Scope Freeze: December 10, 2020 > Code Freeze: December 24, 2020 > Voting Date: January 18, 2021 > Release Date: January 25, 2021 > > > Proposed release scope: > [2] > > > WDYT? > > > [1] > http://apache-ignite-developers.2346864.n4.nabble.com/2-9-1-release-proposal-tp49769p49867.html > [2] > https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.10%20and%20status%20in%20(Resolved%2C%20Closed)%20and%20resolution%20%3D%20Fixed%20order%20by%20priority%20
Re: [DISCUSS] Release Ignite streamer extensions
Igniters, It seems that we came to decision on change of Ignite dependency scopes to "provided". Don't we? The corresponding ticket [1] is ready to final review. Saikat Maitra have already reviewed and approved PR [2]. Could some committer take a final look? [1] - https://issues.apache.org/jira/browse/IGNITE-13634 [2] - https://github.com/apache/ignite-extensions/pull/28 On 28.10.2020 20:24, Saikat Maitra wrote: Hi Alex, Yes, we can start releasing the extensions modules. I would like to also help in releasing a few extensions, it will also help me to be familiarized with the release process. Mikhail and I have been having discussion about whether we should use provided scope for dependencies or shall we use 2.9.0 for ignite.version for the extensions modules? Jira : https://issues.apache.org/jira/browse/IGNITE-13621 PR : https://github.com/apache/ignite-extensions/pull/28/files Please let us know your thoughts. Denis, We have the following jira ticket for documentation work for ignite extensions releases. https://issues.apache.org/jira/browse/IGNITE-12951 Please let us know if the scope in the issue looks good. Regards, Saikat On Wed, Oct 28, 2020 at 10:28 AM Denis Magda wrote: Alex, Should we create a dedicated ticket for the documentation changes or can we reuse IGNITE-13634? As a bare minimum, we need to update maven artifacts' names and versions: https://github.com/apache/ignite/tree/master/docs/_docs/extensions-and-integrations - Denis On Wed, Oct 28, 2020 at 5:10 AM Alexey Goncharuk < alexey.goncha...@gmail.com> wrote: Igniters, Since Ignite 2.9 has been released, I think we can now release the extension modules related to streaming. I noticed that unlike spring autoconfigure, the streamer extensions dependencies do not have provided scope, so I created a ticket to fix that [1]. Anything else we should fix before releasing the extensions? [1] https://issues.apache.org/jira/browse/IGNITE-13634 --AG
[jira] [Created] (IGNITE-13690) Failed to init coordinator caches on concurrent start of nodes with different cache configurations.
Ivan Daschinskiy created IGNITE-13690:
-
Summary: Failed to init coordinator caches on concurrent start of
nodes with different cache configurations.
Key: IGNITE-13690
URL: https://issues.apache.org/jira/browse/IGNITE-13690
Project: Ignite
Issue Type: Bug
Affects Versions: 2.9
Reporter: Ivan Daschinskiy
Scenario:
1. Start simultaneously nodes with different cache configurations
(for simplicity, let client nodes be with configured caches, servers without).
2. When processing first exchange on coordinator, coordinator will fail with
{code:java}
[2020-11-10
13:23:57,232][ERROR][start-node-1][DifferentCacheConfigurationConcurrentStart0]
Got exception while starting (will rollback startup routine).
java.lang.AssertionError: Invalid exchange futures state [cur=6, total=7]
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$17.applyx(CacheAffinitySharedManager.java:1964)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$17.applyx(CacheAffinitySharedManager.java:1935)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.lambda$forAllRegisteredCacheGroups$e0a6939d$1(CacheAffinitySharedManager.java:1265)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11157)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11059)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11039)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.forAllRegisteredCacheGroups(CacheAffinitySharedManager.java:1264)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.initCoordinatorCaches(CacheAffinitySharedManager.java:1935)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCoordinatorCaches(GridDhtPartitionsExchangeFuture.java:716)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:850)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3175)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3021)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
{code}
The main reason is the race on creating {{LocalJoinCachesContext}}, so local
join caches differs from registered caches from other nodes.
Reproducer for zk and ring discoveries are attached.
NB! Not always reproducible -- to increase probability of fail, add sleep in
{{GridDhtPartitionsExchangeFuture#init}}
{code:java}
public void init(boolean newCrd) throws IgniteInterruptedCheckedException {
if (newCrd)
U.sleep(500);
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
