Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Hi everyone, The task of removal IGFS and Hadoop accelerator is ready to review.(https://issues.apache.org/jira/browse/IGNITE-11942) I've already asked some guys to take a look at it but if somebody familiar with this part of code, feel free to take a look at the changes too(especially scripts changes). I also think it is good to decide which release it should be planned on. This task planned for 2.9 right now but I should notice that first of all there are a lot of changes and secondly there are some changes in public API(removed some methods from configuration). So maybe it makes sense to move this ticket to the next release. What do you think? -- Best regards, Anton Kalashnikov 10.02.2020, 15:45, "Alexey Zinoviev" : > Thank you so you much! Will wait:) > > пн, 10 февр. 2020 г. в 15:13, Alexey Goncharuk : > >> Got it, then no need to rush, let's wait for the TF-IGFS decoupling. >> >> пн, 10 февр. 2020 г. в 13:15, Alexey Zinoviev : >> >> > Tensorflow integration uses IGFS, if you have any idea how to store files >> > in memory by another way, please suggest something. >> > I hope to decouple Ignite-TF integration to the separate repository >> before >> > release 2.9 with its own file system over Ignite Caches >> > >> > пн, 10 февр. 2020 г. в 12:49, Ivan Pavlukhin : >> > >> > > Is not it blocked by >> > > https://issues.apache.org/jira/browse/IGNITE-10292 as stated in JIRA? >> > > >> > > @Alex Zinoviev could you please shed some light on this? >> > > >> > > Best regards, >> > > Ivan Pavlukhin >> > > >> > > пн, 10 февр. 2020 г. в 12:46, Anton Kalashnikov : >> > > >> > > > >> > > > I found the correct ticket for such activity - >> > > https://issues.apache.org/jira/browse/IGNITE-11942 >> > > > >> > > > -- >> > > > Best regards, >> > > > Anton Kalashnikov >> > > > >> > > > >> > > > 10.02.2020, 12:16, "Anton Kalashnikov" : >> > > > > Hello. >> > > > > >> > > > > I created a ticket for this activity - >> > > https://issues.apache.org/jira/browse/IGNITE-12647. And if we are >> still >> > > in consensus I'll do it at the nearest time(I've already had the >> prepared >> > > code). >> > > > > >> > > > > -- >> > > > > Best regards, >> > > > > Anton Kalashnikov >> > > > > >> > > > > 10.02.2020, 12:07, "Alexey Goncharuk" > >: >> > > > >> Folks, >> > > > >> >> > > > >> I think there is a consensus here, but we did not remove IGFS >> > > neither in >> > > > >> 2.7 nor in 2.8, did we? Should we schedule a corresponding ticket >> > > for 2.9? >> > > >> >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
We are breaking backwards compatibility, so this can be only done for Ignite 3.0, am I right? On Thu, Jul 9, 2020 at 1:46 PM Anton Kalashnikov wrote: > Hi everyone, > > The task of removal IGFS and Hadoop accelerator is ready to review.( > https://issues.apache.org/jira/browse/IGNITE-11942) > I've already asked some guys to take a look at it but if somebody familiar > with this part of code, feel free to take a look at the changes > too(especially scripts changes). > > I also think it is good to decide which release it should be planned on. > This task planned for 2.9 right now but I should notice that first of all > there are a lot of changes and secondly there are some changes in public > API(removed some methods from configuration). So maybe it makes sense to > move this ticket to the next release. What do you think? > > -- > Best regards, > Anton Kalashnikov > > > 10.02.2020, 15:45, "Alexey Zinoviev" : > > Thank you so you much! Will wait:) > > > > пн, 10 февр. 2020 г. в 15:13, Alexey Goncharuk < > alexey.goncha...@gmail.com>: > > > >> Got it, then no need to rush, let's wait for the TF-IGFS decoupling. > >> > >> пн, 10 февр. 2020 г. в 13:15, Alexey Zinoviev >: > >> > >> > Tensorflow integration uses IGFS, if you have any idea how to store > files > >> > in memory by another way, please suggest something. > >> > I hope to decouple Ignite-TF integration to the separate repository > >> before > >> > release 2.9 with its own file system over Ignite Caches > >> > > >> > пн, 10 февр. 2020 г. в 12:49, Ivan Pavlukhin : > >> > > >> > > Is not it blocked by > >> > > https://issues.apache.org/jira/browse/IGNITE-10292 as stated in > JIRA? > >> > > > >> > > @Alex Zinoviev could you please shed some light on this? > >> > > > >> > > Best regards, > >> > > Ivan Pavlukhin > >> > > > >> > > пн, 10 февр. 2020 г. в 12:46, Anton Kalashnikov >: > >> > > > >> > > > > >> > > > I found the correct ticket for such activity - > >> > > https://issues.apache.org/jira/browse/IGNITE-11942 > >> > > > > >> > > > -- > >> > > > Best regards, > >> > > > Anton Kalashnikov > >> > > > > >> > > > > >> > > > 10.02.2020, 12:16, "Anton Kalashnikov" : > >> > > > > Hello. > >> > > > > > >> > > > > I created a ticket for this activity - > >> > > https://issues.apache.org/jira/browse/IGNITE-12647. And if we are > >> still > >> > > in consensus I'll do it at the nearest time(I've already had the > >> prepared > >> > > code). > >> > > > > > >> > > > > -- > >> > > > > Best regards, > >> > > > > Anton Kalashnikov > >> > > > > > >> > > > > 10.02.2020, 12:07, "Alexey Goncharuk" < > alexey.goncha...@gmail.com > >> >: > >> > > > >> Folks, > >> > > > >> > >> > > > >> I think there is a consensus here, but we did not remove IGFS > >> > > neither in > >> > > > >> 2.7 nor in 2.8, did we? Should we schedule a corresponding > ticket > >> > > for 2.9? > >> > > > >> > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
I don't think it's required to wait until Ignite 3.0 to make this happen. If I'm not mistaken, we stopped releasing Hadoop binaries and sources a long time ago (at least you can't longer find them on the downloads page). Also, we removed all the mentioning from the documentation and website. Nobody complained or requested for a maintenance release since that time. Thus, I would remove the integration in 2.9. If anybody shows up later then they can use the sources in the 2.8 branch and do whatever they want. - Denis On Thu, Jul 9, 2020 at 3:52 AM Pavel Tupitsyn wrote: > We are breaking backwards compatibility, > so this can be only done for Ignite 3.0, am I right? > > On Thu, Jul 9, 2020 at 1:46 PM Anton Kalashnikov > wrote: > > > Hi everyone, > > > > The task of removal IGFS and Hadoop accelerator is ready to review.( > > https://issues.apache.org/jira/browse/IGNITE-11942) > > I've already asked some guys to take a look at it but if somebody > familiar > > with this part of code, feel free to take a look at the changes > > too(especially scripts changes). > > > > I also think it is good to decide which release it should be planned on. > > This task planned for 2.9 right now but I should notice that first of all > > there are a lot of changes and secondly there are some changes in public > > API(removed some methods from configuration). So maybe it makes sense to > > move this ticket to the next release. What do you think? > > > > -- > > Best regards, > > Anton Kalashnikov > > > > > > 10.02.2020, 15:45, "Alexey Zinoviev" : > > > Thank you so you much! Will wait:) > > > > > > пн, 10 февр. 2020 г. в 15:13, Alexey Goncharuk < > > alexey.goncha...@gmail.com>: > > > > > >> Got it, then no need to rush, let's wait for the TF-IGFS decoupling. > > >> > > >> пн, 10 февр. 2020 г. в 13:15, Alexey Zinoviev < > zaleslaw@gmail.com > > >: > > >> > > >> > Tensorflow integration uses IGFS, if you have any idea how to store > > files > > >> > in memory by another way, please suggest something. > > >> > I hope to decouple Ignite-TF integration to the separate repository > > >> before > > >> > release 2.9 with its own file system over Ignite Caches > > >> > > > >> > пн, 10 февр. 2020 г. в 12:49, Ivan Pavlukhin >: > > >> > > > >> > > Is not it blocked by > > >> > > https://issues.apache.org/jira/browse/IGNITE-10292 as stated in > > JIRA? > > >> > > > > >> > > @Alex Zinoviev could you please shed some light on this? > > >> > > > > >> > > Best regards, > > >> > > Ivan Pavlukhin > > >> > > > > >> > > пн, 10 февр. 2020 г. в 12:46, Anton Kalashnikov < > kaa@yandex.ru > > >: > > >> > > > > >> > > > > > >> > > > I found the correct ticket for such activity - > > >> > > https://issues.apache.org/jira/browse/IGNITE-11942 > > >> > > > > > >> > > > -- > > >> > > > Best regards, > > >> > > > Anton Kalashnikov > > >> > > > > > >> > > > > > >> > > > 10.02.2020, 12:16, "Anton Kalashnikov" : > > >> > > > > Hello. > > >> > > > > > > >> > > > > I created a ticket for this activity - > > >> > > https://issues.apache.org/jira/browse/IGNITE-12647. And if we > are > > >> still > > >> > > in consensus I'll do it at the nearest time(I've already had the > > >> prepared > > >> > > code). > > >> > > > > > > >> > > > > -- > > >> > > > > Best regards, > > >> > > > > Anton Kalashnikov > > >> > > > > > > >> > > > > 10.02.2020, 12:07, "Alexey Goncharuk" < > > alexey.goncha...@gmail.com > > >> >: > > >> > > > >> Folks, > > >> > > > >> > > >> > > > >> I think there is a consensus here, but we did not remove > IGFS > > >> > > neither in > > >> > > > >> 2.7 nor in 2.8, did we? Should we schedule a corresponding > > ticket > > >> > > for 2.9? > > >> > > > > >> > > > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Guys, Any updates here? Looks like we still don't have a consensus about release version for this patch (already mention it in the release thread). Currently, the ticket is still targeted to 2.9. ср, 15 июл. 2020 г. в 00:40, Denis Magda : > I don't think it's required to wait until Ignite 3.0 to make this happen. > If I'm not mistaken, we stopped releasing Hadoop binaries and sources a > long time ago (at least you can't longer find them on the downloads page). > Also, we removed all the mentioning from the documentation and website. > Nobody complained or requested for a maintenance release since that time. > Thus, I would remove the integration in 2.9. If anybody shows up later then > they can use the sources in the 2.8 branch and do whatever they want. > > - > Denis > > > On Thu, Jul 9, 2020 at 3:52 AM Pavel Tupitsyn > wrote: > > > We are breaking backwards compatibility, > > so this can be only done for Ignite 3.0, am I right? > > > > On Thu, Jul 9, 2020 at 1:46 PM Anton Kalashnikov > > wrote: > > > > > Hi everyone, > > > > > > The task of removal IGFS and Hadoop accelerator is ready to review.( > > > https://issues.apache.org/jira/browse/IGNITE-11942) > > > I've already asked some guys to take a look at it but if somebody > > familiar > > > with this part of code, feel free to take a look at the changes > > > too(especially scripts changes). > > > > > > I also think it is good to decide which release it should be planned > on. > > > This task planned for 2.9 right now but I should notice that first of > all > > > there are a lot of changes and secondly there are some changes in > public > > > API(removed some methods from configuration). So maybe it makes sense > to > > > move this ticket to the next release. What do you think? > > > > > > -- > > > Best regards, > > > Anton Kalashnikov > > > > > > > > > 10.02.2020, 15:45, "Alexey Zinoviev" : > > > > Thank you so you much! Will wait:) > > > > > > > > пн, 10 февр. 2020 г. в 15:13, Alexey Goncharuk < > > > alexey.goncha...@gmail.com>: > > > > > > > >> Got it, then no need to rush, let's wait for the TF-IGFS > decoupling. > > > >> > > > >> пн, 10 февр. 2020 г. в 13:15, Alexey Zinoviev < > > zaleslaw@gmail.com > > > >: > > > >> > > > >> > Tensorflow integration uses IGFS, if you have any idea how to > store > > > files > > > >> > in memory by another way, please suggest something. > > > >> > I hope to decouple Ignite-TF integration to the separate > repository > > > >> before > > > >> > release 2.9 with its own file system over Ignite Caches > > > >> > > > > >> > пн, 10 февр. 2020 г. в 12:49, Ivan Pavlukhin < > vololo...@gmail.com > > >: > > > >> > > > > >> > > Is not it blocked by > > > >> > > https://issues.apache.org/jira/browse/IGNITE-10292 as stated > in > > > JIRA? > > > >> > > > > > >> > > @Alex Zinoviev could you please shed some light on this? > > > >> > > > > > >> > > Best regards, > > > >> > > Ivan Pavlukhin > > > >> > > > > > >> > > пн, 10 февр. 2020 г. в 12:46, Anton Kalashnikov < > > kaa@yandex.ru > > > >: > > > >> > > > > > >> > > > > > > >> > > > I found the correct ticket for such activity - > > > >> > > https://issues.apache.org/jira/browse/IGNITE-11942 > > > >> > > > > > > >> > > > -- > > > >> > > > Best regards, > > > >> > > > Anton Kalashnikov > > > >> > > > > > > >> > > > > > > >> > > > 10.02.2020, 12:16, "Anton Kalashnikov" : > > > >> > > > > Hello. > > > >> > > > > > > > >> > > > > I created a ticket for this activity - > > > >> > > https://issues.apache.org/jira/browse/IGNITE-12647. And if we > > are > > > >> still > > > >> > > in consensus I'll do it at the nearest time(I've already had > the > > > >> prepared > > > >> > > code). > > > >> > > > > > > > >> > > > > -- > > > >> > > > > Best regards, > > > >> > > > > Anton Kalashnikov > > > >> > > > > > > > >> > > > > 10.02.2020, 12:07, "Alexey Goncharuk" < > > > alexey.goncha...@gmail.com > > > >> >: > > > >> > > > >> Folks, > > > >> > > > >> > > > >> > > > >> I think there is a consensus here, but we did not remove > > IGFS > > > >> > > neither in > > > >> > > > >> 2.7 nor in 2.8, did we? Should we schedule a corresponding > > > ticket > > > >> > > for 2.9? > > > >> > > > > > >> > > > > > > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Hi Alex, It's been a year since we voted to discontinue this integration [1] and it wasn't removed from the source code earlier only because of the internal dependencies with the ML component. Now all the dependencies are gone and Ignite 2.9 is the right version to finish the discontinuation process. It would make sense to wait for Ignite 3.0 only there are some breaking changes in the APIs that will stay in Ignite. @Anton Kalashnikov , you mentioned that you removed some methods from the configuration. Could you please list them here? Are they Hadoop-specific or generic? [1] http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42405.html - Denis On Wed, Jul 22, 2020 at 1:52 AM Alex Plehanov wrote: > Guys, > > Any updates here? Looks like we still don't have a consensus about release > version for this patch (already mention it in the release thread). > Currently, the ticket is still targeted to 2.9. > > ср, 15 июл. 2020 г. в 00:40, Denis Magda : > > > I don't think it's required to wait until Ignite 3.0 to make this happen. > > If I'm not mistaken, we stopped releasing Hadoop binaries and sources a > > long time ago (at least you can't longer find them on the downloads > page). > > Also, we removed all the mentioning from the documentation and website. > > Nobody complained or requested for a maintenance release since that time. > > Thus, I would remove the integration in 2.9. If anybody shows up later > then > > they can use the sources in the 2.8 branch and do whatever they want. > > > > - > > Denis > > > > > > On Thu, Jul 9, 2020 at 3:52 AM Pavel Tupitsyn > > wrote: > > > > > We are breaking backwards compatibility, > > > so this can be only done for Ignite 3.0, am I right? > > > > > > On Thu, Jul 9, 2020 at 1:46 PM Anton Kalashnikov > > > wrote: > > > > > > > Hi everyone, > > > > > > > > The task of removal IGFS and Hadoop accelerator is ready to review.( > > > > https://issues.apache.org/jira/browse/IGNITE-11942) > > > > I've already asked some guys to take a look at it but if somebody > > > familiar > > > > with this part of code, feel free to take a look at the changes > > > > too(especially scripts changes). > > > > > > > > I also think it is good to decide which release it should be planned > > on. > > > > This task planned for 2.9 right now but I should notice that first of > > all > > > > there are a lot of changes and secondly there are some changes in > > public > > > > API(removed some methods from configuration). So maybe it makes sense > > to > > > > move this ticket to the next release. What do you think? > > > > > > > > -- > > > > Best regards, > > > > Anton Kalashnikov > > > > > > > > > > > > 10.02.2020, 15:45, "Alexey Zinoviev" : > > > > > Thank you so you much! Will wait:) > > > > > > > > > > пн, 10 февр. 2020 г. в 15:13, Alexey Goncharuk < > > > > alexey.goncha...@gmail.com>: > > > > > > > > > >> Got it, then no need to rush, let's wait for the TF-IGFS > > decoupling. > > > > >> > > > > >> пн, 10 февр. 2020 г. в 13:15, Alexey Zinoviev < > > > zaleslaw@gmail.com > > > > >: > > > > >> > > > > >> > Tensorflow integration uses IGFS, if you have any idea how to > > store > > > > files > > > > >> > in memory by another way, please suggest something. > > > > >> > I hope to decouple Ignite-TF integration to the separate > > repository > > > > >> before > > > > >> > release 2.9 with its own file system over Ignite Caches > > > > >> > > > > > >> > пн, 10 февр. 2020 г. в 12:49, Ivan Pavlukhin < > > vololo...@gmail.com > > > >: > > > > >> > > > > > >> > > Is not it blocked by > > > > >> > > https://issues.apache.org/jira/browse/IGNITE-10292 as stated > > in > > > > JIRA? > > > > >> > > > > > > >> > > @Alex Zinoviev could you please shed some light on this? > > > > >> > > > > > > >> > > Best regards, > > > > >> > > Ivan Pavlukhin > > > > >> > > > > > > >> > > пн, 10 февр. 2020 г. в 12:46, Anton Kalashnikov < > > > kaa@yandex.ru > > > > >: > > > > >> > > > > > > >> > > > > > > > >> > > > I found the correct ticket for such activity - > > > > >> > > https://issues.apache.org/jira/browse/IGNITE-11942 > > > > >> > > > > > > > >> > > > -- > > > > >> > > > Best regards, > > > > >> > > > Anton Kalashnikov > > > > >> > > > > > > > >> > > > > > > > >> > > > 10.02.2020, 12:16, "Anton Kalashnikov" >: > > > > >> > > > > Hello. > > > > >> > > > > > > > > >> > > > > I created a ticket for this activity - > > > > >> > > https://issues.apache.org/jira/browse/IGNITE-12647. And if > we > > > are > > > > >> still > > > > >> > > in consensus I'll do it at the nearest time(I've already had > > the > > > > >> prepared > > > > >> > > code). > > > > >> > > > > > > > > >> > > > > -- > > > > >> > > > > Best regards, > > > > >> > > > > Anton Kalashnikov > > > > >> > > > > > > > > >> > > > > 10.02.2020, 12:07, "Alexey Goncharuk" < > > > > alexey.goncha...@gmail.com > > > > >> >: > > > > >> > > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Hi, All of these methods are from IgniteConfiguration: Hadoop configuration: - HadoopConfiguration getHadoopConfiguration() - IgniteConfiguration setHadoopConfiguration(HadoopConfiguration hadoopCfg) IGFS (Ignite In-Memory File System) configurations: - FileSystemConfiguration[] getFileSystemConfiguration - IgniteConfiguration setFileSystemConfiguration(FileSystemConfiguration... igfsCfg) thread pool size that will be used to process outgoing IGFS messages: - IgniteConfiguration setIgfsThreadPoolSize(int poolSize) - int getIgfsThreadPoolSize() Of course, I can leave these methods intact but they will be doing nothing so API formally wouldn't be changed but, in fact, features would be removed. Does it make sense? I don't think so and in my opinion, perhaps it is ok to remove these methods right now if we are ready to remove these features right now. (but again, if there are some concerns about it, I can easily to leave these methods with empty implementation) -- Best regards, Anton Kalashnikov 22.07.2020, 17:47, "Denis Magda" : > Hi Alex, > > It's been a year since we voted to discontinue this integration [1] and it > wasn't removed from the source code earlier only because of the internal > dependencies with the ML component. Now all the dependencies are gone and > Ignite 2.9 is the right version to finish the discontinuation process. It > would make sense to wait for Ignite 3.0 only there are some breaking > changes in the APIs that will stay in Ignite. > > @Anton Kalashnikov , you mentioned that you > removed some methods from the configuration. Could you please list them > here? Are they Hadoop-specific or generic? > > [1] > http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42405.html > > - > Denis > > On Wed, Jul 22, 2020 at 1:52 AM Alex Plehanov > wrote: > >> Guys, >> >> Any updates here? Looks like we still don't have a consensus about release >> version for this patch (already mention it in the release thread). >> Currently, the ticket is still targeted to 2.9. >> >> ср, 15 июл. 2020 г. в 00:40, Denis Magda : >> >> > I don't think it's required to wait until Ignite 3.0 to make this happen. >> > If I'm not mistaken, we stopped releasing Hadoop binaries and sources a >> > long time ago (at least you can't longer find them on the downloads >> page). >> > Also, we removed all the mentioning from the documentation and website. >> > Nobody complained or requested for a maintenance release since that time. >> > Thus, I would remove the integration in 2.9. If anybody shows up later >> then >> > they can use the sources in the 2.8 branch and do whatever they want. >> > >> > - >> > Denis >> > >> > >> > On Thu, Jul 9, 2020 at 3:52 AM Pavel Tupitsyn >> > wrote: >> > >> > > We are breaking backwards compatibility, >> > > so this can be only done for Ignite 3.0, am I right? >> > > >> > > On Thu, Jul 9, 2020 at 1:46 PM Anton Kalashnikov >> > > wrote: >> > > >> > > > Hi everyone, >> > > > >> > > > The task of removal IGFS and Hadoop accelerator is ready to review.( >> > > > https://issues.apache.org/jira/browse/IGNITE-11942) >> > > > I've already asked some guys to take a look at it but if somebody >> > > familiar >> > > > with this part of code, feel free to take a look at the changes >> > > > too(especially scripts changes). >> > > > >> > > > I also think it is good to decide which release it should be planned >> > on. >> > > > This task planned for 2.9 right now but I should notice that first of >> > all >> > > > there are a lot of changes and secondly there are some changes in >> > public >> > > > API(removed some methods from configuration). So maybe it makes sense >> > to >> > > > move this ticket to the next release. What do you think? >> > > > >> > > > -- >> > > > Best regards, >> > > > Anton Kalashnikov >> > > > >> > > > >> > > > 10.02.2020, 15:45, "Alexey Zinoviev" : >> > > > > Thank you so you much! Will wait:) >> > > > > >> > > > > пн, 10 февр. 2020 г. в 15:13, Alexey Goncharuk < >> > > > alexey.goncha...@gmail.com>: >> > > > > >> > > > >> Got it, then no need to rush, let's wait for the TF-IGFS >> > decoupling. >> > > > >> >> > > > >> пн, 10 февр. 2020 г. в 13:15, Alexey Zinoviev < >> > > zaleslaw@gmail.com >> > > > >: >> > > > >> >> > > > >> > Tensorflow integration uses IGFS, if you have any idea how to >> > store >> > > > files >> > > > >> > in memory by another way, please suggest something. >> > > > >> > I hope to decouple Ignite-TF integration to the separate >> > repository >> > > > >> before >> > > > >> > release 2.9 with its own file system over Ignite Caches >> > > > >> > >> > > > >> > пн, 10 февр. 2020 г. в 12:49, Ivan Pavlukhin < >> > vololo...@gmail.com >> > > >: >> > > > >> > >> > > > >> > > Is not it blocked by >> > > > >> > > https://issues.apache.org/jira/browse/IGNITE-10292 as stated >> > in >> > > > JIRA? >> > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Sorry, I was mistaken, we can not leave these methods because at least FileSystemConfiguration and HadoopConfiguration require corresponded classes that were deleted. So I think we should just remove it right now. -- Best regards, Anton Kalashnikov 22.07.2020, 18:56, "Anton Kalashnikov" : > Hi, > > All of these methods are from IgniteConfiguration: > Hadoop configuration: > - HadoopConfiguration getHadoopConfiguration() > - IgniteConfiguration setHadoopConfiguration(HadoopConfiguration hadoopCfg) > > IGFS (Ignite In-Memory File System) configurations: > - FileSystemConfiguration[] getFileSystemConfiguration > - IgniteConfiguration setFileSystemConfiguration(FileSystemConfiguration... > igfsCfg) > > thread pool size that will be used to process outgoing IGFS messages: > - IgniteConfiguration setIgfsThreadPoolSize(int poolSize) > - int getIgfsThreadPoolSize() > > Of course, I can leave these methods intact but they will be doing nothing so > API formally wouldn't be changed but, in fact, features would be removed. > Does it make sense? I don't think so and in my opinion, perhaps it is ok to > remove these methods right now if we are ready to remove these features right > now. (but again, if there are some concerns about it, I can easily to leave > these methods with empty implementation) > > -- > Best regards, > Anton Kalashnikov > > 22.07.2020, 17:47, "Denis Magda" : >> Hi Alex, >> >> It's been a year since we voted to discontinue this integration [1] and it >> wasn't removed from the source code earlier only because of the internal >> dependencies with the ML component. Now all the dependencies are gone and >> Ignite 2.9 is the right version to finish the discontinuation process. It >> would make sense to wait for Ignite 3.0 only there are some breaking >> changes in the APIs that will stay in Ignite. >> >> @Anton Kalashnikov , you mentioned that you >> removed some methods from the configuration. Could you please list them >> here? Are they Hadoop-specific or generic? >> >> [1] >> >> http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42405.html >> >> - >> Denis >> >> On Wed, Jul 22, 2020 at 1:52 AM Alex Plehanov >> wrote: >> >>> Guys, >>> >>> Any updates here? Looks like we still don't have a consensus about release >>> version for this patch (already mention it in the release thread). >>> Currently, the ticket is still targeted to 2.9. >>> >>> ср, 15 июл. 2020 г. в 00:40, Denis Magda : >>> >>> > I don't think it's required to wait until Ignite 3.0 to make this >>> happen. >>> > If I'm not mistaken, we stopped releasing Hadoop binaries and sources a >>> > long time ago (at least you can't longer find them on the downloads >>> page). >>> > Also, we removed all the mentioning from the documentation and website. >>> > Nobody complained or requested for a maintenance release since that >>> time. >>> > Thus, I would remove the integration in 2.9. If anybody shows up later >>> then >>> > they can use the sources in the 2.8 branch and do whatever they want. >>> > >>> > - >>> > Denis >>> > >>> > >>> > On Thu, Jul 9, 2020 at 3:52 AM Pavel Tupitsyn >>> > wrote: >>> > >>> > > We are breaking backwards compatibility, >>> > > so this can be only done for Ignite 3.0, am I right? >>> > > >>> > > On Thu, Jul 9, 2020 at 1:46 PM Anton Kalashnikov >>> > > wrote: >>> > > >>> > > > Hi everyone, >>> > > > >>> > > > The task of removal IGFS and Hadoop accelerator is ready to review.( >>> > > > https://issues.apache.org/jira/browse/IGNITE-11942) >>> > > > I've already asked some guys to take a look at it but if somebody >>> > > familiar >>> > > > with this part of code, feel free to take a look at the changes >>> > > > too(especially scripts changes). >>> > > > >>> > > > I also think it is good to decide which release it should be planned >>> > on. >>> > > > This task planned for 2.9 right now but I should notice that first >>> of >>> > all >>> > > > there are a lot of changes and secondly there are some changes in >>> > public >>> > > > API(removed some methods from configuration). So maybe it makes >>> sense >>> > to >>> > > > move this ticket to the next release. What do you think? >>> > > > >>> > > > -- >>> > > > Best regards, >>> > > > Anton Kalashnikov >>> > > > >>> > > > >>> > > > 10.02.2020, 15:45, "Alexey Zinoviev" : >>> > > > > Thank you so you much! Will wait:) >>> > > > > >>> > > > > пн, 10 февр. 2020 г. в 15:13, Alexey Goncharuk < >>> > > > alexey.goncha...@gmail.com>: >>> > > > > >>> > > > >> Got it, then no need to rush, let's wait for the TF-IGFS >>> > decoupling. >>> > > > >> >>> > > > >> пн, 10 февр. 2020 г. в 13:15, Alexey Zinoviev < >>> > > zaleslaw@gmail.com >>> > > > >: >>> > > > >> >>> > > > >> > Tensorflow integration uses IGFS, if you have any idea how to >>> > store
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Hi, Could you please help me to unsubscribe all these ignite mails. I have unsubscribed it, but still I am getting lot of emails Could you please help me.. On Wed, 22 Jul, 2020, 9:45 pm Anton Kalashnikov, wrote: > Sorry, I was mistaken, we can not leave these methods because at least > FileSystemConfiguration and HadoopConfiguration require corresponded > classes that were deleted. So I think we should just remove it right now. > > -- > Best regards, > Anton Kalashnikov > > > > 22.07.2020, 18:56, "Anton Kalashnikov" : > > Hi, > > > > All of these methods are from IgniteConfiguration: > > Hadoop configuration: > > - HadoopConfiguration getHadoopConfiguration() > > - IgniteConfiguration setHadoopConfiguration(HadoopConfiguration > hadoopCfg) > > > > IGFS (Ignite In-Memory File System) configurations: > > - FileSystemConfiguration[] getFileSystemConfiguration > > - IgniteConfiguration > setFileSystemConfiguration(FileSystemConfiguration... igfsCfg) > > > > thread pool size that will be used to process outgoing IGFS messages: > > - IgniteConfiguration setIgfsThreadPoolSize(int poolSize) > > - int getIgfsThreadPoolSize() > > > > Of course, I can leave these methods intact but they will be doing > nothing so API formally wouldn't be changed but, in fact, features would be > removed. Does it make sense? I don't think so and in my opinion, perhaps it > is ok to remove these methods right now if we are ready to remove these > features right now. (but again, if there are some concerns about it, I can > easily to leave these methods with empty implementation) > > > > -- > > Best regards, > > Anton Kalashnikov > > > > 22.07.2020, 17:47, "Denis Magda" : > >> Hi Alex, > >> > >> It's been a year since we voted to discontinue this integration [1] > and it > >> wasn't removed from the source code earlier only because of the > internal > >> dependencies with the ML component. Now all the dependencies are gone > and > >> Ignite 2.9 is the right version to finish the discontinuation process. > It > >> would make sense to wait for Ignite 3.0 only there are some breaking > >> changes in the APIs that will stay in Ignite. > >> > >> @Anton Kalashnikov , you mentioned that > you > >> removed some methods from the configuration. Could you please list them > >> here? Are they Hadoop-specific or generic? > >> > >> [1] > >> > http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42405.html > >> > >> - > >> Denis > >> > >> On Wed, Jul 22, 2020 at 1:52 AM Alex Plehanov > > >> wrote: > >> > >>> Guys, > >>> > >>> Any updates here? Looks like we still don't have a consensus about > release > >>> version for this patch (already mention it in the release thread). > >>> Currently, the ticket is still targeted to 2.9. > >>> > >>> ср, 15 июл. 2020 г. в 00:40, Denis Magda : > >>> > >>> > I don't think it's required to wait until Ignite 3.0 to make this > happen. > >>> > If I'm not mistaken, we stopped releasing Hadoop binaries and > sources a > >>> > long time ago (at least you can't longer find them on the downloads > >>> page). > >>> > Also, we removed all the mentioning from the documentation and > website. > >>> > Nobody complained or requested for a maintenance release since > that time. > >>> > Thus, I would remove the integration in 2.9. If anybody shows up > later > >>> then > >>> > they can use the sources in the 2.8 branch and do whatever they > want. > >>> > > >>> > - > >>> > Denis > >>> > > >>> > > >>> > On Thu, Jul 9, 2020 at 3:52 AM Pavel Tupitsyn < > ptupit...@apache.org> > >>> > wrote: > >>> > > >>> > > We are breaking backwards compatibility, > >>> > > so this can be only done for Ignite 3.0, am I right? > >>> > > > >>> > > On Thu, Jul 9, 2020 at 1:46 PM Anton Kalashnikov < > kaa@yandex.ru> > >>> > > wrote: > >>> > > > >>> > > > Hi everyone, > >>> > > > > >>> > > > The task of removal IGFS and Hadoop accelerator is ready to > review.( > >>> > > > https://issues.apache.org/jira/browse/IGNITE-11942) > >>> > > > I've already asked some guys to take a look at it but if > somebody > >>> > > familiar > >>> > > > with this part of code, feel free to take a look at the changes > >>> > > > too(especially scripts changes). > >>> > > > > >>> > > > I also think it is good to decide which release it should be > planned > >>> > on. > >>> > > > This task planned for 2.9 right now but I should notice that > first of > >>> > all > >>> > > > there are a lot of changes and secondly there are some changes > in > >>> > public > >>> > > > API(removed some methods from configuration). So maybe it > makes sense > >>> > to > >>> > > > move this ticket to the next release. What do you think? > >>> > > > > >>> > > > -- > >>> > > > Best regards, > >>> > > > Anton Kalashnikov > >>> > > > > >>> > > > > >>> > > > 10.02.2020, 15:45, "Alexey Zinoviev" : > >>> > > > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
The removal of these methods will not impact anybody during migration to 2.9. Unless you use the Hadoop Accelerator or IGFS in your application. In the latter scenario, well, the integration has already been discontinued and you need to clean your application code before moving forward to 2.9. Anyway, it's highly unlikely that we have any users of the integrations, otherwise, they would show up throughout the year after learning that the integration is no longer supported and discontinued. Pavel, do you still prefer us merging the changes in Ignite 3.0 or agree to do that in 2.9? - Denis On Wed, Jul 22, 2020 at 8:56 AM Anton Kalashnikov wrote: > Hi, > > All of these methods are from IgniteConfiguration: > Hadoop configuration: > - HadoopConfiguration getHadoopConfiguration() > - IgniteConfiguration setHadoopConfiguration(HadoopConfiguration hadoopCfg) > > IGFS (Ignite In-Memory File System) configurations: > - FileSystemConfiguration[] getFileSystemConfiguration > - IgniteConfiguration > setFileSystemConfiguration(FileSystemConfiguration... igfsCfg) > > thread pool size that will be used to process outgoing IGFS messages: > - IgniteConfiguration setIgfsThreadPoolSize(int poolSize) > - int getIgfsThreadPoolSize() > > > Of course, I can leave these methods intact but they will be doing nothing > so API formally wouldn't be changed but, in fact, features would be > removed. Does it make sense? I don't think so and in my opinion, perhaps it > is ok to remove these methods right now if we are ready to remove these > features right now. (but again, if there are some concerns about it, I can > easily to leave these methods with empty implementation) > > -- > Best regards, > Anton Kalashnikov > > > > 22.07.2020, 17:47, "Denis Magda" : > > Hi Alex, > > > > It's been a year since we voted to discontinue this integration [1] and > it > > wasn't removed from the source code earlier only because of the internal > > dependencies with the ML component. Now all the dependencies are gone and > > Ignite 2.9 is the right version to finish the discontinuation process. It > > would make sense to wait for Ignite 3.0 only there are some breaking > > changes in the APIs that will stay in Ignite. > > > > @Anton Kalashnikov , you mentioned that you > > removed some methods from the configuration. Could you please list them > > here? Are they Hadoop-specific or generic? > > > > [1] > > > http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42405.html > > > > - > > Denis > > > > On Wed, Jul 22, 2020 at 1:52 AM Alex Plehanov > > wrote: > > > >> Guys, > >> > >> Any updates here? Looks like we still don't have a consensus about > release > >> version for this patch (already mention it in the release thread). > >> Currently, the ticket is still targeted to 2.9. > >> > >> ср, 15 июл. 2020 г. в 00:40, Denis Magda : > >> > >> > I don't think it's required to wait until Ignite 3.0 to make this > happen. > >> > If I'm not mistaken, we stopped releasing Hadoop binaries and > sources a > >> > long time ago (at least you can't longer find them on the downloads > >> page). > >> > Also, we removed all the mentioning from the documentation and > website. > >> > Nobody complained or requested for a maintenance release since that > time. > >> > Thus, I would remove the integration in 2.9. If anybody shows up > later > >> then > >> > they can use the sources in the 2.8 branch and do whatever they want. > >> > > >> > - > >> > Denis > >> > > >> > > >> > On Thu, Jul 9, 2020 at 3:52 AM Pavel Tupitsyn > >> > wrote: > >> > > >> > > We are breaking backwards compatibility, > >> > > so this can be only done for Ignite 3.0, am I right? > >> > > > >> > > On Thu, Jul 9, 2020 at 1:46 PM Anton Kalashnikov < > kaa@yandex.ru> > >> > > wrote: > >> > > > >> > > > Hi everyone, > >> > > > > >> > > > The task of removal IGFS and Hadoop accelerator is ready to > review.( > >> > > > https://issues.apache.org/jira/browse/IGNITE-11942) > >> > > > I've already asked some guys to take a look at it but if somebody > >> > > familiar > >> > > > with this part of code, feel free to take a look at the changes > >> > > > too(especially scripts changes). > >> > > > > >> > > > I also think it is good to decide which release it should be > planned > >> > on. > >> > > > This task planned for 2.9 right now but I should notice that > first of > >> > all > >> > > > there are a lot of changes and secondly there are some changes in > >> > public > >> > > > API(removed some methods from configuration). So maybe it makes > sense > >> > to > >> > > > move this ticket to the next release. What do you think? > >> > > > > >> > > > -- > >> > > > Best regards, > >> > > > Anton Kalashnikov > >> > > > > >> > > > > >> > > > 10.02.2020, 15:45, "Alexey Zinoviev" : > >> > > > > Thank you so you much! Will wait:) > >> > > > > > >> > > > > пн, 10 февр. 2020 г. в 15:13, Alexey
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Hi, Send an email to this address: dev-unsubscr...@ignite.apache.org - Denis On Wed, Jul 22, 2020 at 9:28 AM C Ravikiran wrote: > Hi, > > Could you please help me to unsubscribe all these ignite mails. > > I have unsubscribed it, but still I am getting lot of emails > Could you please help me.. > > On Wed, 22 Jul, 2020, 9:45 pm Anton Kalashnikov, > wrote: > > > Sorry, I was mistaken, we can not leave these methods because at least > > FileSystemConfiguration and HadoopConfiguration require corresponded > > classes that were deleted. So I think we should just remove it right now. > > > > -- > > Best regards, > > Anton Kalashnikov > > > > > > > > 22.07.2020, 18:56, "Anton Kalashnikov" : > > > Hi, > > > > > > All of these methods are from IgniteConfiguration: > > > Hadoop configuration: > > > - HadoopConfiguration getHadoopConfiguration() > > > - IgniteConfiguration setHadoopConfiguration(HadoopConfiguration > > hadoopCfg) > > > > > > IGFS (Ignite In-Memory File System) configurations: > > > - FileSystemConfiguration[] getFileSystemConfiguration > > > - IgniteConfiguration > > setFileSystemConfiguration(FileSystemConfiguration... igfsCfg) > > > > > > thread pool size that will be used to process outgoing IGFS messages: > > > - IgniteConfiguration setIgfsThreadPoolSize(int poolSize) > > > - int getIgfsThreadPoolSize() > > > > > > Of course, I can leave these methods intact but they will be doing > > nothing so API formally wouldn't be changed but, in fact, features would > be > > removed. Does it make sense? I don't think so and in my opinion, perhaps > it > > is ok to remove these methods right now if we are ready to remove these > > features right now. (but again, if there are some concerns about it, I > can > > easily to leave these methods with empty implementation) > > > > > > -- > > > Best regards, > > > Anton Kalashnikov > > > > > > 22.07.2020, 17:47, "Denis Magda" : > > >> Hi Alex, > > >> > > >> It's been a year since we voted to discontinue this integration [1] > > and it > > >> wasn't removed from the source code earlier only because of the > > internal > > >> dependencies with the ML component. Now all the dependencies are gone > > and > > >> Ignite 2.9 is the right version to finish the discontinuation > process. > > It > > >> would make sense to wait for Ignite 3.0 only there are some breaking > > >> changes in the APIs that will stay in Ignite. > > >> > > >> @Anton Kalashnikov , you mentioned that > > you > > >> removed some methods from the configuration. Could you please list > them > > >> here? Are they Hadoop-specific or generic? > > >> > > >> [1] > > >> > > > http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42405.html > > >> > > >> - > > >> Denis > > >> > > >> On Wed, Jul 22, 2020 at 1:52 AM Alex Plehanov < > plehanov.a...@gmail.com > > > > > >> wrote: > > >> > > >>> Guys, > > >>> > > >>> Any updates here? Looks like we still don't have a consensus about > > release > > >>> version for this patch (already mention it in the release thread). > > >>> Currently, the ticket is still targeted to 2.9. > > >>> > > >>> ср, 15 июл. 2020 г. в 00:40, Denis Magda : > > >>> > > >>> > I don't think it's required to wait until Ignite 3.0 to make this > > happen. > > >>> > If I'm not mistaken, we stopped releasing Hadoop binaries and > > sources a > > >>> > long time ago (at least you can't longer find them on the > downloads > > >>> page). > > >>> > Also, we removed all the mentioning from the documentation and > > website. > > >>> > Nobody complained or requested for a maintenance release since > > that time. > > >>> > Thus, I would remove the integration in 2.9. If anybody shows up > > later > > >>> then > > >>> > they can use the sources in the 2.8 branch and do whatever they > > want. > > >>> > > > >>> > - > > >>> > Denis > > >>> > > > >>> > > > >>> > On Thu, Jul 9, 2020 at 3:52 AM Pavel Tupitsyn < > > ptupit...@apache.org> > > >>> > wrote: > > >>> > > > >>> > > We are breaking backwards compatibility, > > >>> > > so this can be only done for Ignite 3.0, am I right? > > >>> > > > > >>> > > On Thu, Jul 9, 2020 at 1:46 PM Anton Kalashnikov < > > kaa@yandex.ru> > > >>> > > wrote: > > >>> > > > > >>> > > > Hi everyone, > > >>> > > > > > >>> > > > The task of removal IGFS and Hadoop accelerator is ready to > > review.( > > >>> > > > https://issues.apache.org/jira/browse/IGNITE-11942) > > >>> > > > I've already asked some guys to take a look at it but if > > somebody > > >>> > > familiar > > >>> > > > with this part of code, feel free to take a look at the > changes > > >>> > > > too(especially scripts changes). > > >>> > > > > > >>> > > > I also think it is good to decide which release it should be > > planned > > >>> > on. > > >>> > > > This task planned for 2.9 right now but I should notice that > > first of > > >>> > all > > >>>
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Denis, Thanks for the explanation, I don't have any objections anymore. On Wed, Jul 22, 2020 at 7:46 PM Denis Magda wrote: > Hi, > > Send an email to this address: dev-unsubscr...@ignite.apache.org > > - > Denis > > > On Wed, Jul 22, 2020 at 9:28 AM C Ravikiran > wrote: > > > Hi, > > > > Could you please help me to unsubscribe all these ignite mails. > > > > I have unsubscribed it, but still I am getting lot of emails > > Could you please help me.. > > > > On Wed, 22 Jul, 2020, 9:45 pm Anton Kalashnikov, > > wrote: > > > > > Sorry, I was mistaken, we can not leave these methods because at least > > > FileSystemConfiguration and HadoopConfiguration require corresponded > > > classes that were deleted. So I think we should just remove it right > now. > > > > > > -- > > > Best regards, > > > Anton Kalashnikov > > > > > > > > > > > > 22.07.2020, 18:56, "Anton Kalashnikov" : > > > > Hi, > > > > > > > > All of these methods are from IgniteConfiguration: > > > > Hadoop configuration: > > > > - HadoopConfiguration getHadoopConfiguration() > > > > - IgniteConfiguration setHadoopConfiguration(HadoopConfiguration > > > hadoopCfg) > > > > > > > > IGFS (Ignite In-Memory File System) configurations: > > > > - FileSystemConfiguration[] getFileSystemConfiguration > > > > - IgniteConfiguration > > > setFileSystemConfiguration(FileSystemConfiguration... igfsCfg) > > > > > > > > thread pool size that will be used to process outgoing IGFS messages: > > > > - IgniteConfiguration setIgfsThreadPoolSize(int poolSize) > > > > - int getIgfsThreadPoolSize() > > > > > > > > Of course, I can leave these methods intact but they will be doing > > > nothing so API formally wouldn't be changed but, in fact, features > would > > be > > > removed. Does it make sense? I don't think so and in my opinion, > perhaps > > it > > > is ok to remove these methods right now if we are ready to remove these > > > features right now. (but again, if there are some concerns about it, I > > can > > > easily to leave these methods with empty implementation) > > > > > > > > -- > > > > Best regards, > > > > Anton Kalashnikov > > > > > > > > 22.07.2020, 17:47, "Denis Magda" : > > > >> Hi Alex, > > > >> > > > >> It's been a year since we voted to discontinue this integration [1] > > > and it > > > >> wasn't removed from the source code earlier only because of the > > > internal > > > >> dependencies with the ML component. Now all the dependencies are > gone > > > and > > > >> Ignite 2.9 is the right version to finish the discontinuation > > process. > > > It > > > >> would make sense to wait for Ignite 3.0 only there are some > breaking > > > >> changes in the APIs that will stay in Ignite. > > > >> > > > >> @Anton Kalashnikov , you mentioned > that > > > you > > > >> removed some methods from the configuration. Could you please list > > them > > > >> here? Are they Hadoop-specific or generic? > > > >> > > > >> [1] > > > >> > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42405.html > > > >> > > > >> - > > > >> Denis > > > >> > > > >> On Wed, Jul 22, 2020 at 1:52 AM Alex Plehanov < > > plehanov.a...@gmail.com > > > > > > > >> wrote: > > > >> > > > >>> Guys, > > > >>> > > > >>> Any updates here? Looks like we still don't have a consensus > about > > > release > > > >>> version for this patch (already mention it in the release > thread). > > > >>> Currently, the ticket is still targeted to 2.9. > > > >>> > > > >>> ср, 15 июл. 2020 г. в 00:40, Denis Magda : > > > >>> > > > >>> > I don't think it's required to wait until Ignite 3.0 to make > this > > > happen. > > > >>> > If I'm not mistaken, we stopped releasing Hadoop binaries and > > > sources a > > > >>> > long time ago (at least you can't longer find them on the > > downloads > > > >>> page). > > > >>> > Also, we removed all the mentioning from the documentation and > > > website. > > > >>> > Nobody complained or requested for a maintenance release since > > > that time. > > > >>> > Thus, I would remove the integration in 2.9. If anybody shows > up > > > later > > > >>> then > > > >>> > they can use the sources in the 2.8 branch and do whatever they > > > want. > > > >>> > > > > >>> > - > > > >>> > Denis > > > >>> > > > > >>> > > > > >>> > On Thu, Jul 9, 2020 at 3:52 AM Pavel Tupitsyn < > > > ptupit...@apache.org> > > > >>> > wrote: > > > >>> > > > > >>> > > We are breaking backwards compatibility, > > > >>> > > so this can be only done for Ignite 3.0, am I right? > > > >>> > > > > > >>> > > On Thu, Jul 9, 2020 at 1:46 PM Anton Kalashnikov < > > > kaa@yandex.ru> > > > >>> > > wrote: > > > >>> > > > > > >>> > > > Hi everyone, > > > >>> > > > > > > >>> > > > The task of removal IGFS and Hadoop accelerator is ready to > > > review.( > > > >>> > > > https://issues.apache.org/jira/browse/IGNITE-11942) > > > >>> > > > I've already asked some guys
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Thanks, Pavel! Anton, Alex, we're in agreement to merge the changes in Ignite 2.9. Please go ahead removing the Hadoop/IGFS modules from both the master and 2.9 branches. - Denis On Thu, Jul 23, 2020 at 8:14 AM Pavel Tupitsyn wrote: > Denis, > > Thanks for the explanation, I don't have any objections anymore. > > On Wed, Jul 22, 2020 at 7:46 PM Denis Magda wrote: > > > Hi, > > > > Send an email to this address: dev-unsubscr...@ignite.apache.org > > > > - > > Denis > > > > > > On Wed, Jul 22, 2020 at 9:28 AM C Ravikiran > > wrote: > > > > > Hi, > > > > > > Could you please help me to unsubscribe all these ignite mails. > > > > > > I have unsubscribed it, but still I am getting lot of emails > > > Could you please help me.. > > > > > > On Wed, 22 Jul, 2020, 9:45 pm Anton Kalashnikov, > > > wrote: > > > > > > > Sorry, I was mistaken, we can not leave these methods because at > least > > > > FileSystemConfiguration and HadoopConfiguration require corresponded > > > > classes that were deleted. So I think we should just remove it right > > now. > > > > > > > > -- > > > > Best regards, > > > > Anton Kalashnikov > > > > > > > > > > > > > > > > 22.07.2020, 18:56, "Anton Kalashnikov" : > > > > > Hi, > > > > > > > > > > All of these methods are from IgniteConfiguration: > > > > > Hadoop configuration: > > > > > - HadoopConfiguration getHadoopConfiguration() > > > > > - IgniteConfiguration setHadoopConfiguration(HadoopConfiguration > > > > hadoopCfg) > > > > > > > > > > IGFS (Ignite In-Memory File System) configurations: > > > > > - FileSystemConfiguration[] getFileSystemConfiguration > > > > > - IgniteConfiguration > > > > setFileSystemConfiguration(FileSystemConfiguration... igfsCfg) > > > > > > > > > > thread pool size that will be used to process outgoing IGFS > messages: > > > > > - IgniteConfiguration setIgfsThreadPoolSize(int poolSize) > > > > > - int getIgfsThreadPoolSize() > > > > > > > > > > Of course, I can leave these methods intact but they will be doing > > > > nothing so API formally wouldn't be changed but, in fact, features > > would > > > be > > > > removed. Does it make sense? I don't think so and in my opinion, > > perhaps > > > it > > > > is ok to remove these methods right now if we are ready to remove > these > > > > features right now. (but again, if there are some concerns about it, > I > > > can > > > > easily to leave these methods with empty implementation) > > > > > > > > > > -- > > > > > Best regards, > > > > > Anton Kalashnikov > > > > > > > > > > 22.07.2020, 17:47, "Denis Magda" : > > > > >> Hi Alex, > > > > >> > > > > >> It's been a year since we voted to discontinue this integration > [1] > > > > and it > > > > >> wasn't removed from the source code earlier only because of the > > > > internal > > > > >> dependencies with the ML component. Now all the dependencies are > > gone > > > > and > > > > >> Ignite 2.9 is the right version to finish the discontinuation > > > process. > > > > It > > > > >> would make sense to wait for Ignite 3.0 only there are some > > breaking > > > > >> changes in the APIs that will stay in Ignite. > > > > >> > > > > >> @Anton Kalashnikov , you mentioned > > that > > > > you > > > > >> removed some methods from the configuration. Could you please > list > > > them > > > > >> here? Are they Hadoop-specific or generic? > > > > >> > > > > >> [1] > > > > >> > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42405.html > > > > >> > > > > >> - > > > > >> Denis > > > > >> > > > > >> On Wed, Jul 22, 2020 at 1:52 AM Alex Plehanov < > > > plehanov.a...@gmail.com > > > > > > > > > >> wrote: > > > > >> > > > > >>> Guys, > > > > >>> > > > > >>> Any updates here? Looks like we still don't have a consensus > > about > > > > release > > > > >>> version for this patch (already mention it in the release > > thread). > > > > >>> Currently, the ticket is still targeted to 2.9. > > > > >>> > > > > >>> ср, 15 июл. 2020 г. в 00:40, Denis Magda : > > > > >>> > > > > >>> > I don't think it's required to wait until Ignite 3.0 to make > > this > > > > happen. > > > > >>> > If I'm not mistaken, we stopped releasing Hadoop binaries and > > > > sources a > > > > >>> > long time ago (at least you can't longer find them on the > > > downloads > > > > >>> page). > > > > >>> > Also, we removed all the mentioning from the documentation > and > > > > website. > > > > >>> > Nobody complained or requested for a maintenance release > since > > > > that time. > > > > >>> > Thus, I would remove the integration in 2.9. If anybody shows > > up > > > > later > > > > >>> then > > > > >>> > they can use the sources in the 2.8 branch and do whatever > they > > > > want. > > > > >>> > > > > > >>> > - > > > > >>> > Denis > > > > >>> > > > > > >>> > > > > > >>> > On Thu, Jul 9, 2020 at 3:52 AM Pavel Tupitsyn < > > > > ptupit...@apache.org> > > > > >>> > wro
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Folks, I think there is a consensus here, but we did not remove IGFS neither in 2.7 nor in 2.8, did we? Should we schedule a corresponding ticket for 2.9?
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Hello. I created a ticket for this activity - https://issues.apache.org/jira/browse/IGNITE-12647. And if we are still in consensus I'll do it at the nearest time(I've already had the prepared code). -- Best regards, Anton Kalashnikov 10.02.2020, 12:07, "Alexey Goncharuk" : > Folks, > > I think there is a consensus here, but we did not remove IGFS neither in > 2.7 nor in 2.8, did we? Should we schedule a corresponding ticket for 2.9?
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
I found the correct ticket for such activity - https://issues.apache.org/jira/browse/IGNITE-11942 -- Best regards, Anton Kalashnikov 10.02.2020, 12:16, "Anton Kalashnikov" : > Hello. > > I created a ticket for this activity - > https://issues.apache.org/jira/browse/IGNITE-12647. And if we are still in > consensus I'll do it at the nearest time(I've already had the prepared code). > > -- > Best regards, > Anton Kalashnikov > > 10.02.2020, 12:07, "Alexey Goncharuk" : >> Folks, >> >> I think there is a consensus here, but we did not remove IGFS neither in >> 2.7 nor in 2.8, did we? Should we schedule a corresponding ticket for 2.9?
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Is not it blocked by https://issues.apache.org/jira/browse/IGNITE-10292 as stated in JIRA? @Alex Zinoviev could you please shed some light on this? Best regards, Ivan Pavlukhin пн, 10 февр. 2020 г. в 12:46, Anton Kalashnikov : > > I found the correct ticket for such activity - > https://issues.apache.org/jira/browse/IGNITE-11942 > > -- > Best regards, > Anton Kalashnikov > > > 10.02.2020, 12:16, "Anton Kalashnikov" : > > Hello. > > > > I created a ticket for this activity - > > https://issues.apache.org/jira/browse/IGNITE-12647. And if we are still in > > consensus I'll do it at the nearest time(I've already had the prepared > > code). > > > > -- > > Best regards, > > Anton Kalashnikov > > > > 10.02.2020, 12:07, "Alexey Goncharuk" : > >> Folks, > >> > >> I think there is a consensus here, but we did not remove IGFS neither in > >> 2.7 nor in 2.8, did we? Should we schedule a corresponding ticket for 2.9?
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Tensorflow integration uses IGFS, if you have any idea how to store files in memory by another way, please suggest something. I hope to decouple Ignite-TF integration to the separate repository before release 2.9 with its own file system over Ignite Caches пн, 10 февр. 2020 г. в 12:49, Ivan Pavlukhin : > Is not it blocked by > https://issues.apache.org/jira/browse/IGNITE-10292 as stated in JIRA? > > @Alex Zinoviev could you please shed some light on this? > > Best regards, > Ivan Pavlukhin > > пн, 10 февр. 2020 г. в 12:46, Anton Kalashnikov : > > > > > I found the correct ticket for such activity - > https://issues.apache.org/jira/browse/IGNITE-11942 > > > > -- > > Best regards, > > Anton Kalashnikov > > > > > > 10.02.2020, 12:16, "Anton Kalashnikov" : > > > Hello. > > > > > > I created a ticket for this activity - > https://issues.apache.org/jira/browse/IGNITE-12647. And if we are still > in consensus I'll do it at the nearest time(I've already had the prepared > code). > > > > > > -- > > > Best regards, > > > Anton Kalashnikov > > > > > > 10.02.2020, 12:07, "Alexey Goncharuk" : > > >> Folks, > > >> > > >> I think there is a consensus here, but we did not remove IGFS > neither in > > >> 2.7 nor in 2.8, did we? Should we schedule a corresponding ticket > for 2.9? >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Got it, then no need to rush, let's wait for the TF-IGFS decoupling. пн, 10 февр. 2020 г. в 13:15, Alexey Zinoviev : > Tensorflow integration uses IGFS, if you have any idea how to store files > in memory by another way, please suggest something. > I hope to decouple Ignite-TF integration to the separate repository before > release 2.9 with its own file system over Ignite Caches > > пн, 10 февр. 2020 г. в 12:49, Ivan Pavlukhin : > > > Is not it blocked by > > https://issues.apache.org/jira/browse/IGNITE-10292 as stated in JIRA? > > > > @Alex Zinoviev could you please shed some light on this? > > > > Best regards, > > Ivan Pavlukhin > > > > пн, 10 февр. 2020 г. в 12:46, Anton Kalashnikov : > > > > > > > > I found the correct ticket for such activity - > > https://issues.apache.org/jira/browse/IGNITE-11942 > > > > > > -- > > > Best regards, > > > Anton Kalashnikov > > > > > > > > > 10.02.2020, 12:16, "Anton Kalashnikov" : > > > > Hello. > > > > > > > > I created a ticket for this activity - > > https://issues.apache.org/jira/browse/IGNITE-12647. And if we are still > > in consensus I'll do it at the nearest time(I've already had the prepared > > code). > > > > > > > > -- > > > > Best regards, > > > > Anton Kalashnikov > > > > > > > > 10.02.2020, 12:07, "Alexey Goncharuk" : > > > >> Folks, > > > >> > > > >> I think there is a consensus here, but we did not remove IGFS > > neither in > > > >> 2.7 nor in 2.8, did we? Should we schedule a corresponding ticket > > for 2.9? > > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Thank you so you much! Will wait:) пн, 10 февр. 2020 г. в 15:13, Alexey Goncharuk : > Got it, then no need to rush, let's wait for the TF-IGFS decoupling. > > пн, 10 февр. 2020 г. в 13:15, Alexey Zinoviev : > > > Tensorflow integration uses IGFS, if you have any idea how to store files > > in memory by another way, please suggest something. > > I hope to decouple Ignite-TF integration to the separate repository > before > > release 2.9 with its own file system over Ignite Caches > > > > пн, 10 февр. 2020 г. в 12:49, Ivan Pavlukhin : > > > > > Is not it blocked by > > > https://issues.apache.org/jira/browse/IGNITE-10292 as stated in JIRA? > > > > > > @Alex Zinoviev could you please shed some light on this? > > > > > > Best regards, > > > Ivan Pavlukhin > > > > > > пн, 10 февр. 2020 г. в 12:46, Anton Kalashnikov : > > > > > > > > > > > I found the correct ticket for such activity - > > > https://issues.apache.org/jira/browse/IGNITE-11942 > > > > > > > > -- > > > > Best regards, > > > > Anton Kalashnikov > > > > > > > > > > > > 10.02.2020, 12:16, "Anton Kalashnikov" : > > > > > Hello. > > > > > > > > > > I created a ticket for this activity - > > > https://issues.apache.org/jira/browse/IGNITE-12647. And if we are > still > > > in consensus I'll do it at the nearest time(I've already had the > prepared > > > code). > > > > > > > > > > -- > > > > > Best regards, > > > > > Anton Kalashnikov > > > > > > > > > > 10.02.2020, 12:07, "Alexey Goncharuk" >: > > > > >> Folks, > > > > >> > > > > >> I think there is a consensus here, but we did not remove IGFS > > > neither in > > > > >> 2.7 nor in 2.8, did we? Should we schedule a corresponding ticket > > > for 2.9? > > > > > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Denis, I must say that aforementioned solutions for a Hadoop ecosystem appear from time to time in questions on a user mailing list. So, it seems that there is a practical need for such solutions. But of course it does not mean that we should continue a support of IGFS and Hadoop Accelerator. If both are not solutions that fit well common use cases then we should discontinue it. If any of them is very good for it's purposes but we do not have a capacity to support it without sacrificing main Ignite goals then we still should discontinue it in my mind. P.S. Personally I am a fan of UNIX way. I like ideas of a single responsibility and integrations. And I suppose there are other Ignite features which could be dropped. ср, 12 июн. 2019 г. в 21:04, Denis Magda : > > Igniters, > > I'd like us to move on and finish our conversation on the IGFS [1] and > Hadoop Accelerator [2] support. > > To my knowledge, there is no single committer who maintains the > integrations; they are no longer tested and, even more, the community > stopped providing the binaries since Ignite 2.6.0 release (look for > In-Memory Hadoop Accelerator table [3]). > > Why all of that happened? Because of a little value, something succeeds > while something fails. Does it mean that Ignite cannot be used for Hadoop > acceleration, in general? No, quite the opposite, it CAN be used, but a > solution is different. Have Ignite with native persistence deployed close > to your Hadoop cluster (replace GridGain with Ignite) [4]. > > So, I propose we remove IGFS and In-Memory Hadoop Accelerator from our > master repository and rework existing public documentation showing how to > achieve the acceleration with Ignite. > > Any supporters or objections? > > > [1] https://apacheignite-fs.readme.io/docs/in-memory-file-system > [2] https://apacheignite-fs.readme.io/docs/hadoop-accelerator > [3] https://ignite.apache.org/download.cgi#binaries > [4] > https://docs.gridgain.com/docs/bdb-getting-started#section-gridgain-data-lake-accelerator > > - > Denis -- Best regards, Ivan Pavlukhin
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Denis, I fully support this idea. First, looking back, I do not think it was a good design in the first place to build IGFS on top of Ignite caches. Second, I have never seen a case where IGFS provided significant performance boost. Usually it's either all data already fits buffer cache, and IGFS caching is not needed; or data does not fit buffer cache, and access pattern is close to full scan and additional caching in IGFS does not make sense. пн, 17 июн. 2019 г. в 11:28, Павлухин Иван : > Denis, > > I must say that aforementioned solutions for a Hadoop ecosystem appear > from time to time in questions on a user mailing list. So, it seems > that there is a practical need for such solutions. > > But of course it does not mean that we should continue a support of > IGFS and Hadoop Accelerator. If both are not solutions that fit well > common use cases then we should discontinue it. If any of them is very > good for it's purposes but we do not have a capacity to support it > without sacrificing main Ignite goals then we still should discontinue > it in my mind. > > P.S. Personally I am a fan of UNIX way. I like ideas of a single > responsibility and integrations. And I suppose there are other Ignite > features which could be dropped. > > ср, 12 июн. 2019 г. в 21:04, Denis Magda : > > > > > Igniters, > > > > I'd like us to move on and finish our conversation on the IGFS [1] and > > Hadoop Accelerator [2] support. > > > > To my knowledge, there is no single committer who maintains the > > integrations; they are no longer tested and, even more, the community > > stopped providing the binaries since Ignite 2.6.0 release (look for > > In-Memory Hadoop Accelerator table [3]). > > > > Why all of that happened? Because of a little value, something succeeds > > while something fails. Does it mean that Ignite cannot be used for Hadoop > > acceleration, in general? No, quite the opposite, it CAN be used, but a > > solution is different. Have Ignite with native persistence deployed close > > to your Hadoop cluster (replace GridGain with Ignite) [4]. > > > > So, I propose we remove IGFS and In-Memory Hadoop Accelerator from our > > master repository and rework existing public documentation showing how to > > achieve the acceleration with Ignite. > > > > Any supporters or objections? > > > > > > [1] https://apacheignite-fs.readme.io/docs/in-memory-file-system > > [2] https://apacheignite-fs.readme.io/docs/hadoop-accelerator > > [3] https://ignite.apache.org/download.cgi#binaries > > [4] > > > https://docs.gridgain.com/docs/bdb-getting-started#section-gridgain-data-lake-accelerator > > > > - > > Denis > > > > -- > Best regards, > Ivan Pavlukhin >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
+1 from me to reduce supported feature list. Guys, are we talking about Ignite 3.0? В Пн, 17/06/2019 в 11:57 +0300, Alexey Goncharuk пишет: > Denis, > > I fully support this idea. > > First, looking back, I do not think it was a good design in the first place > to build IGFS on top of Ignite caches. Second, I have never seen a case > where IGFS provided significant performance boost. Usually it's either all > data already fits buffer cache, and IGFS caching is not needed; or data > does not fit buffer cache, and access pattern is close to full scan and > additional caching in IGFS does not make sense. > > пн, 17 июн. 2019 г. в 11:28, Павлухин Иван : > > > Denis, > > > > I must say that aforementioned solutions for a Hadoop ecosystem appear > > from time to time in questions on a user mailing list. So, it seems > > that there is a practical need for such solutions. > > > > But of course it does not mean that we should continue a support of > > IGFS and Hadoop Accelerator. If both are not solutions that fit well > > common use cases then we should discontinue it. If any of them is very > > good for it's purposes but we do not have a capacity to support it > > without sacrificing main Ignite goals then we still should discontinue > > it in my mind. > > > > P.S. Personally I am a fan of UNIX way. I like ideas of a single > > responsibility and integrations. And I suppose there are other Ignite > > features which could be dropped. > > > > ср, 12 июн. 2019 г. в 21:04, Denis Magda : > > > > > > > > Igniters, > > > > > > I'd like us to move on and finish our conversation on the IGFS [1] and > > > Hadoop Accelerator [2] support. > > > > > > To my knowledge, there is no single committer who maintains the > > > integrations; they are no longer tested and, even more, the community > > > stopped providing the binaries since Ignite 2.6.0 release (look for > > > In-Memory Hadoop Accelerator table [3]). > > > > > > Why all of that happened? Because of a little value, something succeeds > > > while something fails. Does it mean that Ignite cannot be used for Hadoop > > > acceleration, in general? No, quite the opposite, it CAN be used, but a > > > solution is different. Have Ignite with native persistence deployed close > > > to your Hadoop cluster (replace GridGain with Ignite) [4]. > > > > > > So, I propose we remove IGFS and In-Memory Hadoop Accelerator from our > > > master repository and rework existing public documentation showing how to > > > achieve the acceleration with Ignite. > > > > > > Any supporters or objections? > > > > > > > > > [1] https://apacheignite-fs.readme.io/docs/in-memory-file-system > > > [2] https://apacheignite-fs.readme.io/docs/hadoop-accelerator > > > [3] https://ignite.apache.org/download.cgi#binaries > > > [4] > > > > > > > https://docs.gridgain.com/docs/bdb-getting-started#section-gridgain-data-lake-accelerator > > > > > > - > > > Denis > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > > signature.asc Description: This is a digitally signed message part
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
+1 from me. Hadoop Accelerator seems like an outdated solution, and I believe IGFS was only added to support Hadoop Accelerator. Best Regards, Igor On Mon, Jun 17, 2019 at 12:00 PM Nikolay Izhikov wrote: > +1 from me to reduce supported feature list. > > Guys, are we talking about Ignite 3.0? > > > В Пн, 17/06/2019 в 11:57 +0300, Alexey Goncharuk пишет: > > Denis, > > > > I fully support this idea. > > > > First, looking back, I do not think it was a good design in the first > place > > to build IGFS on top of Ignite caches. Second, I have never seen a case > > where IGFS provided significant performance boost. Usually it's either > all > > data already fits buffer cache, and IGFS caching is not needed; or data > > does not fit buffer cache, and access pattern is close to full scan and > > additional caching in IGFS does not make sense. > > > > пн, 17 июн. 2019 г. в 11:28, Павлухин Иван : > > > > > Denis, > > > > > > I must say that aforementioned solutions for a Hadoop ecosystem appear > > > from time to time in questions on a user mailing list. So, it seems > > > that there is a practical need for such solutions. > > > > > > But of course it does not mean that we should continue a support of > > > IGFS and Hadoop Accelerator. If both are not solutions that fit well > > > common use cases then we should discontinue it. If any of them is very > > > good for it's purposes but we do not have a capacity to support it > > > without sacrificing main Ignite goals then we still should discontinue > > > it in my mind. > > > > > > P.S. Personally I am a fan of UNIX way. I like ideas of a single > > > responsibility and integrations. And I suppose there are other Ignite > > > features which could be dropped. > > > > > > ср, 12 июн. 2019 г. в 21:04, Denis Magda : > > > > > > > > > > > Igniters, > > > > > > > > I'd like us to move on and finish our conversation on the IGFS [1] > and > > > > Hadoop Accelerator [2] support. > > > > > > > > To my knowledge, there is no single committer who maintains the > > > > integrations; they are no longer tested and, even more, the community > > > > stopped providing the binaries since Ignite 2.6.0 release (look for > > > > In-Memory Hadoop Accelerator table [3]). > > > > > > > > Why all of that happened? Because of a little value, something > succeeds > > > > while something fails. Does it mean that Ignite cannot be used for > Hadoop > > > > acceleration, in general? No, quite the opposite, it CAN be used, > but a > > > > solution is different. Have Ignite with native persistence deployed > close > > > > to your Hadoop cluster (replace GridGain with Ignite) [4]. > > > > > > > > So, I propose we remove IGFS and In-Memory Hadoop Accelerator from > our > > > > master repository and rework existing public documentation showing > how to > > > > achieve the acceleration with Ignite. > > > > > > > > Any supporters or objections? > > > > > > > > > > > > [1] https://apacheignite-fs.readme.io/docs/in-memory-file-system > > > > [2] https://apacheignite-fs.readme.io/docs/hadoop-accelerator > > > > [3] https://ignite.apache.org/download.cgi#binaries > > > > [4] > > > > > > > > > > > https://docs.gridgain.com/docs/bdb-getting-started#section-gridgain-data-lake-accelerator > > > > > > > > - > > > > Denis > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Hi Igniters, +1 from me provided that we move sources to some supplementary repository. If someone later would like to maintain/fix this module, he/she should be able to access sources with current state of the master. Sincerely, Dmitriy Pavlov пн, 17 июн. 2019 г. в 18:25, Igor Sapego : > +1 from me. Hadoop Accelerator seems like an outdated solution, > and I believe IGFS was only added to support Hadoop Accelerator. > > Best Regards, > Igor > > > On Mon, Jun 17, 2019 at 12:00 PM Nikolay Izhikov > wrote: > > > +1 from me to reduce supported feature list. > > > > Guys, are we talking about Ignite 3.0? > > > > > > В Пн, 17/06/2019 в 11:57 +0300, Alexey Goncharuk пишет: > > > Denis, > > > > > > I fully support this idea. > > > > > > First, looking back, I do not think it was a good design in the first > > place > > > to build IGFS on top of Ignite caches. Second, I have never seen a case > > > where IGFS provided significant performance boost. Usually it's either > > all > > > data already fits buffer cache, and IGFS caching is not needed; or data > > > does not fit buffer cache, and access pattern is close to full scan and > > > additional caching in IGFS does not make sense. > > > > > > пн, 17 июн. 2019 г. в 11:28, Павлухин Иван : > > > > > > > Denis, > > > > > > > > I must say that aforementioned solutions for a Hadoop ecosystem > appear > > > > from time to time in questions on a user mailing list. So, it seems > > > > that there is a practical need for such solutions. > > > > > > > > But of course it does not mean that we should continue a support of > > > > IGFS and Hadoop Accelerator. If both are not solutions that fit well > > > > common use cases then we should discontinue it. If any of them is > very > > > > good for it's purposes but we do not have a capacity to support it > > > > without sacrificing main Ignite goals then we still should > discontinue > > > > it in my mind. > > > > > > > > P.S. Personally I am a fan of UNIX way. I like ideas of a single > > > > responsibility and integrations. And I suppose there are other Ignite > > > > features which could be dropped. > > > > > > > > ср, 12 июн. 2019 г. в 21:04, Denis Magda : > > > > > > > > > > > > > > Igniters, > > > > > > > > > > I'd like us to move on and finish our conversation on the IGFS [1] > > and > > > > > Hadoop Accelerator [2] support. > > > > > > > > > > To my knowledge, there is no single committer who maintains the > > > > > integrations; they are no longer tested and, even more, the > community > > > > > stopped providing the binaries since Ignite 2.6.0 release (look for > > > > > In-Memory Hadoop Accelerator table [3]). > > > > > > > > > > Why all of that happened? Because of a little value, something > > succeeds > > > > > while something fails. Does it mean that Ignite cannot be used for > > Hadoop > > > > > acceleration, in general? No, quite the opposite, it CAN be used, > > but a > > > > > solution is different. Have Ignite with native persistence deployed > > close > > > > > to your Hadoop cluster (replace GridGain with Ignite) [4]. > > > > > > > > > > So, I propose we remove IGFS and In-Memory Hadoop Accelerator from > > our > > > > > master repository and rework existing public documentation showing > > how to > > > > > achieve the acceleration with Ignite. > > > > > > > > > > Any supporters or objections? > > > > > > > > > > > > > > > [1] https://apacheignite-fs.readme.io/docs/in-memory-file-system > > > > > [2] https://apacheignite-fs.readme.io/docs/hadoop-accelerator > > > > > [3] https://ignite.apache.org/download.cgi#binaries > > > > > [4] > > > > > > > > > > > > > > > > https://docs.gridgain.com/docs/bdb-getting-started#section-gridgain-data-lake-accelerator > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > > -- > > > > Best regards, > > > > Ivan Pavlukhin > > > > > > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
> > +1 from me to reduce supported feature list. > Guys, are we talking about Ignite 3.0? Nikolay, I would discontinue IGFS before 3.0. Let's do this in the next release? As for other features and integrations, 3.0 should be considered as a version. > +1 from me provided that we move sources to some supplementary repository. > If someone later would like to maintain/fix this module, he/she should be > able to access sources with current state of the master. Dmitry, are you suggesting to move the sources to Github and abandon them there? Sort of legacy code cemetery. - Denis On Mon, Jun 17, 2019 at 2:00 AM Nikolay Izhikov wrote: > +1 from me to reduce supported feature list. > > Guys, are we talking about Ignite 3.0? > > > В Пн, 17/06/2019 в 11:57 +0300, Alexey Goncharuk пишет: > > Denis, > > > > I fully support this idea. > > > > First, looking back, I do not think it was a good design in the first > place > > to build IGFS on top of Ignite caches. Second, I have never seen a case > > where IGFS provided significant performance boost. Usually it's either > all > > data already fits buffer cache, and IGFS caching is not needed; or data > > does not fit buffer cache, and access pattern is close to full scan and > > additional caching in IGFS does not make sense. > > > > пн, 17 июн. 2019 г. в 11:28, Павлухин Иван : > > > > > Denis, > > > > > > I must say that aforementioned solutions for a Hadoop ecosystem appear > > > from time to time in questions on a user mailing list. So, it seems > > > that there is a practical need for such solutions. > > > > > > But of course it does not mean that we should continue a support of > > > IGFS and Hadoop Accelerator. If both are not solutions that fit well > > > common use cases then we should discontinue it. If any of them is very > > > good for it's purposes but we do not have a capacity to support it > > > without sacrificing main Ignite goals then we still should discontinue > > > it in my mind. > > > > > > P.S. Personally I am a fan of UNIX way. I like ideas of a single > > > responsibility and integrations. And I suppose there are other Ignite > > > features which could be dropped. > > > > > > ср, 12 июн. 2019 г. в 21:04, Denis Magda : > > > > > > > > > > > Igniters, > > > > > > > > I'd like us to move on and finish our conversation on the IGFS [1] > and > > > > Hadoop Accelerator [2] support. > > > > > > > > To my knowledge, there is no single committer who maintains the > > > > integrations; they are no longer tested and, even more, the community > > > > stopped providing the binaries since Ignite 2.6.0 release (look for > > > > In-Memory Hadoop Accelerator table [3]). > > > > > > > > Why all of that happened? Because of a little value, something > succeeds > > > > while something fails. Does it mean that Ignite cannot be used for > Hadoop > > > > acceleration, in general? No, quite the opposite, it CAN be used, > but a > > > > solution is different. Have Ignite with native persistence deployed > close > > > > to your Hadoop cluster (replace GridGain with Ignite) [4]. > > > > > > > > So, I propose we remove IGFS and In-Memory Hadoop Accelerator from > our > > > > master repository and rework existing public documentation showing > how to > > > > achieve the acceleration with Ignite. > > > > > > > > Any supporters or objections? > > > > > > > > > > > > [1] https://apacheignite-fs.readme.io/docs/in-memory-file-system > > > > [2] https://apacheignite-fs.readme.io/docs/hadoop-accelerator > > > > [3] https://ignite.apache.org/download.cgi#binaries > > > > [4] > > > > > > > > > > > https://docs.gridgain.com/docs/bdb-getting-started#section-gridgain-data-lake-accelerator > > > > > > > > - > > > > Denis > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Folks, +1 to reduce the number of supported features. Probably, the best solution will be removing IGFS from core module and making it as an Ignite plugin (will require some efforts to do this). I've also think we can move IGFS to the separate branch (from the master one) if someone will decide merge to latest changes from the master branch to build Ignite from scratch with IGFS feature. On Mon, 17 Jun 2019 at 22:42, Denis Magda wrote: > > > > > +1 from me to reduce supported feature list. > > Guys, are we talking about Ignite 3.0? > > > Nikolay, I would discontinue IGFS before 3.0. Let's do this in the next > release? As for other features and integrations, 3.0 should be considered > as a version. > > > > +1 from me provided that we move sources to some supplementary repository. > > If someone later would like to maintain/fix this module, he/she should be > > able to access sources with current state of the master. > > > Dmitry, are you suggesting to move the sources to Github and abandon them > there? Sort of legacy code cemetery. > > - > Denis > > > On Mon, Jun 17, 2019 at 2:00 AM Nikolay Izhikov wrote: > > > +1 from me to reduce supported feature list. > > > > Guys, are we talking about Ignite 3.0? > > > > > > В Пн, 17/06/2019 в 11:57 +0300, Alexey Goncharuk пишет: > > > Denis, > > > > > > I fully support this idea. > > > > > > First, looking back, I do not think it was a good design in the first > > place > > > to build IGFS on top of Ignite caches. Second, I have never seen a case > > > where IGFS provided significant performance boost. Usually it's either > > all > > > data already fits buffer cache, and IGFS caching is not needed; or data > > > does not fit buffer cache, and access pattern is close to full scan and > > > additional caching in IGFS does not make sense. > > > > > > пн, 17 июн. 2019 г. в 11:28, Павлухин Иван : > > > > > > > Denis, > > > > > > > > I must say that aforementioned solutions for a Hadoop ecosystem appear > > > > from time to time in questions on a user mailing list. So, it seems > > > > that there is a practical need for such solutions. > > > > > > > > But of course it does not mean that we should continue a support of > > > > IGFS and Hadoop Accelerator. If both are not solutions that fit well > > > > common use cases then we should discontinue it. If any of them is very > > > > good for it's purposes but we do not have a capacity to support it > > > > without sacrificing main Ignite goals then we still should discontinue > > > > it in my mind. > > > > > > > > P.S. Personally I am a fan of UNIX way. I like ideas of a single > > > > responsibility and integrations. And I suppose there are other Ignite > > > > features which could be dropped. > > > > > > > > ср, 12 июн. 2019 г. в 21:04, Denis Magda : > > > > > > > > > > > > > > Igniters, > > > > > > > > > > I'd like us to move on and finish our conversation on the IGFS [1] > > and > > > > > Hadoop Accelerator [2] support. > > > > > > > > > > To my knowledge, there is no single committer who maintains the > > > > > integrations; they are no longer tested and, even more, the community > > > > > stopped providing the binaries since Ignite 2.6.0 release (look for > > > > > In-Memory Hadoop Accelerator table [3]). > > > > > > > > > > Why all of that happened? Because of a little value, something > > succeeds > > > > > while something fails. Does it mean that Ignite cannot be used for > > Hadoop > > > > > acceleration, in general? No, quite the opposite, it CAN be used, > > but a > > > > > solution is different. Have Ignite with native persistence deployed > > close > > > > > to your Hadoop cluster (replace GridGain with Ignite) [4]. > > > > > > > > > > So, I propose we remove IGFS and In-Memory Hadoop Accelerator from > > our > > > > > master repository and rework existing public documentation showing > > how to > > > > > achieve the acceleration with Ignite. > > > > > > > > > > Any supporters or objections? > > > > > > > > > > > > > > > [1] https://apacheignite-fs.readme.io/docs/in-memory-file-system > > > > > [2] https://apacheignite-fs.readme.io/docs/hadoop-accelerator > > > > > [3] https://ignite.apache.org/download.cgi#binaries > > > > > [4] > > > > > > > > > > > > > > > https://docs.gridgain.com/docs/bdb-getting-started#section-gridgain-data-lake-accelerator > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > > -- > > > > Best regards, > > > > Ivan Pavlukhin > > > > > >
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Igniters, Thanks a lot for sharing your opinion. As I see, there is a consensus that IGFS and Hadoop Accelerator are to be discontinued and no longer supported by the community. As for the source code, if the community prefers moving the source code to another repository rather than removing it, then let's do it. I see 3 solutions here: - The simplest - just point out to the latest Ignite release branch that has the source code. This should be Ignite 2.6.0. Remove from Ignite master. - Decouple from the master and move to a 3rd party Github repo. More complicated and time-consuming. - See if we should move the component to Apache Attic ( http://attic.apache.org): the Attic is designed for projects to be retired but not for the components. Thus, that might be not an option. Personally, I'm for the first approach. Does it sound reasonable? - Denis On Tue, Jun 18, 2019 at 7:39 AM Maxim Muzafarov wrote: > Folks, > > +1 to reduce the number of supported features. > > Probably, the best solution will be removing IGFS from core module and > making it as an Ignite plugin (will require some efforts to do this). > I've also think we can move IGFS to the separate branch (from the > master one) if someone will decide merge to latest changes from the > master branch to build Ignite from scratch with IGFS feature. > > On Mon, 17 Jun 2019 at 22:42, Denis Magda wrote: > > > > > > > > +1 from me to reduce supported feature list. > > > Guys, are we talking about Ignite 3.0? > > > > > > Nikolay, I would discontinue IGFS before 3.0. Let's do this in the next > > release? As for other features and integrations, 3.0 should be considered > > as a version. > > > > > > > +1 from me provided that we move sources to some supplementary > repository. > > > If someone later would like to maintain/fix this module, he/she should > be > > > able to access sources with current state of the master. > > > > > > Dmitry, are you suggesting to move the sources to Github and abandon them > > there? Sort of legacy code cemetery. > > > > - > > Denis > > > > > > On Mon, Jun 17, 2019 at 2:00 AM Nikolay Izhikov > wrote: > > > > > +1 from me to reduce supported feature list. > > > > > > Guys, are we talking about Ignite 3.0? > > > > > > > > > В Пн, 17/06/2019 в 11:57 +0300, Alexey Goncharuk пишет: > > > > Denis, > > > > > > > > I fully support this idea. > > > > > > > > First, looking back, I do not think it was a good design in the first > > > place > > > > to build IGFS on top of Ignite caches. Second, I have never seen a > case > > > > where IGFS provided significant performance boost. Usually it's > either > > > all > > > > data already fits buffer cache, and IGFS caching is not needed; or > data > > > > does not fit buffer cache, and access pattern is close to full scan > and > > > > additional caching in IGFS does not make sense. > > > > > > > > пн, 17 июн. 2019 г. в 11:28, Павлухин Иван : > > > > > > > > > Denis, > > > > > > > > > > I must say that aforementioned solutions for a Hadoop ecosystem > appear > > > > > from time to time in questions on a user mailing list. So, it seems > > > > > that there is a practical need for such solutions. > > > > > > > > > > But of course it does not mean that we should continue a support of > > > > > IGFS and Hadoop Accelerator. If both are not solutions that fit > well > > > > > common use cases then we should discontinue it. If any of them is > very > > > > > good for it's purposes but we do not have a capacity to support it > > > > > without sacrificing main Ignite goals then we still should > discontinue > > > > > it in my mind. > > > > > > > > > > P.S. Personally I am a fan of UNIX way. I like ideas of a single > > > > > responsibility and integrations. And I suppose there are other > Ignite > > > > > features which could be dropped. > > > > > > > > > > ср, 12 июн. 2019 г. в 21:04, Denis Magda : > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > I'd like us to move on and finish our conversation on the IGFS > [1] > > > and > > > > > > Hadoop Accelerator [2] support. > > > > > > > > > > > > To my knowledge, there is no single committer who maintains the > > > > > > integrations; they are no longer tested and, even more, the > community > > > > > > stopped providing the binaries since Ignite 2.6.0 release (look > for > > > > > > In-Memory Hadoop Accelerator table [3]). > > > > > > > > > > > > Why all of that happened? Because of a little value, something > > > succeeds > > > > > > while something fails. Does it mean that Ignite cannot be used > for > > > Hadoop > > > > > > acceleration, in general? No, quite the opposite, it CAN be used, > > > but a > > > > > > solution is different. Have Ignite with native persistence > deployed > > > close > > > > > > to your Hadoop cluster (replace GridGain with Ignite) [4]. > > > > > > > > > > > > So, I propose we remove IGFS and In-Memory Hadoop Accelerator > from > > > our > > > > > > master repository and rework exis
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Hi Denis, I like the first approach with applying Maxim's idea of creating a branch named -igfs-Hadoop (not release, but current master state). 2nd) 3rd party repo can be Apache repo just like ignite-release. But it's true it is time-consuming to move code. 3rd) Attic is for projects, I hope no one here wants to Ignite to be there :) I'm not sure it is possible to move just one component there. But if it is possible, we should anyway start from 2nd option and create standalone repo ignite-igfs-hadoop (and it will become later attic-ignite-igfs-hadoop). But if someone could stand up and say he/she wants to do migration from one repo to another (option 2), I like it as well. Sincerely вт, 18 июн. 2019 г. в 21:05, Denis Magda : > Igniters, > > Thanks a lot for sharing your opinion. As I see, there is a consensus that > IGFS and Hadoop Accelerator are to be discontinued and no longer supported > by the community. > > As for the source code, if the community prefers moving the source code to > another repository rather than removing it, then let's do it. I see 3 > solutions here: > >- The simplest - just point out to the latest Ignite release branch that >has the source code. This should be Ignite 2.6.0. Remove from Ignite > master. >- Decouple from the master and move to a 3rd party Github repo. More >complicated and time-consuming. >- See if we should move the component to Apache Attic ( >http://attic.apache.org): the Attic is designed for projects to be >retired but not for the components. Thus, that might be not an option. > > Personally, I'm for the first approach. Does it sound reasonable? > > - > Denis > > > On Tue, Jun 18, 2019 at 7:39 AM Maxim Muzafarov > wrote: > > > Folks, > > > > +1 to reduce the number of supported features. > > > > Probably, the best solution will be removing IGFS from core module and > > making it as an Ignite plugin (will require some efforts to do this). > > I've also think we can move IGFS to the separate branch (from the > > master one) if someone will decide merge to latest changes from the > > master branch to build Ignite from scratch with IGFS feature. > > > > On Mon, 17 Jun 2019 at 22:42, Denis Magda wrote: > > > > > > > > > > > +1 from me to reduce supported feature list. > > > > Guys, are we talking about Ignite 3.0? > > > > > > > > > Nikolay, I would discontinue IGFS before 3.0. Let's do this in the next > > > release? As for other features and integrations, 3.0 should be > considered > > > as a version. > > > > > > > > > > +1 from me provided that we move sources to some supplementary > > repository. > > > > If someone later would like to maintain/fix this module, he/she > should > > be > > > > able to access sources with current state of the master. > > > > > > > > > Dmitry, are you suggesting to move the sources to Github and abandon > them > > > there? Sort of legacy code cemetery. > > > > > > - > > > Denis > > > > > > > > > On Mon, Jun 17, 2019 at 2:00 AM Nikolay Izhikov > > wrote: > > > > > > > +1 from me to reduce supported feature list. > > > > > > > > Guys, are we talking about Ignite 3.0? > > > > > > > > > > > > В Пн, 17/06/2019 в 11:57 +0300, Alexey Goncharuk пишет: > > > > > Denis, > > > > > > > > > > I fully support this idea. > > > > > > > > > > First, looking back, I do not think it was a good design in the > first > > > > place > > > > > to build IGFS on top of Ignite caches. Second, I have never seen a > > case > > > > > where IGFS provided significant performance boost. Usually it's > > either > > > > all > > > > > data already fits buffer cache, and IGFS caching is not needed; or > > data > > > > > does not fit buffer cache, and access pattern is close to full scan > > and > > > > > additional caching in IGFS does not make sense. > > > > > > > > > > пн, 17 июн. 2019 г. в 11:28, Павлухин Иван : > > > > > > > > > > > Denis, > > > > > > > > > > > > I must say that aforementioned solutions for a Hadoop ecosystem > > appear > > > > > > from time to time in questions on a user mailing list. So, it > seems > > > > > > that there is a practical need for such solutions. > > > > > > > > > > > > But of course it does not mean that we should continue a support > of > > > > > > IGFS and Hadoop Accelerator. If both are not solutions that fit > > well > > > > > > common use cases then we should discontinue it. If any of them is > > very > > > > > > good for it's purposes but we do not have a capacity to support > it > > > > > > without sacrificing main Ignite goals then we still should > > discontinue > > > > > > it in my mind. > > > > > > > > > > > > P.S. Personally I am a fan of UNIX way. I like ideas of a single > > > > > > responsibility and integrations. And I suppose there are other > > Ignite > > > > > > features which could be dropped. > > > > > > > > > > > > ср, 12 июн. 2019 г. в 21:04, Denis Magda : > > > > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > I'd like us to move on and finish our conve
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Dmitriy, I like the first approach with applying Maxim's idea of creating a branch > named -igfs-Hadoop (not release, but current master state). +1 for this approach. Any other opinions before we finalize this discussion? - Denis On Tue, Jun 18, 2019 at 11:28 AM Dmitriy Pavlov wrote: > Hi Denis, > > I like the first approach with applying Maxim's idea of creating a branch > named -igfs-Hadoop (not release, but current master state). > 2nd) 3rd party repo can be Apache repo just like ignite-release. But it's > true it is time-consuming to move code. > 3rd) Attic is for projects, I hope no one here wants to Ignite to be there > :) I'm not sure it is possible to move just one component there. But if it > is possible, we should anyway start from 2nd option and create standalone > repo ignite-igfs-hadoop (and it will become later > attic-ignite-igfs-hadoop). > > But if someone could stand up and say he/she wants to do migration from one > repo to another (option 2), I like it as well. > > Sincerely > > вт, 18 июн. 2019 г. в 21:05, Denis Magda : > > > Igniters, > > > > Thanks a lot for sharing your opinion. As I see, there is a consensus > that > > IGFS and Hadoop Accelerator are to be discontinued and no longer > supported > > by the community. > > > > As for the source code, if the community prefers moving the source code > to > > another repository rather than removing it, then let's do it. I see 3 > > solutions here: > > > >- The simplest - just point out to the latest Ignite release branch > that > >has the source code. This should be Ignite 2.6.0. Remove from Ignite > > master. > >- Decouple from the master and move to a 3rd party Github repo. More > >complicated and time-consuming. > >- See if we should move the component to Apache Attic ( > >http://attic.apache.org): the Attic is designed for projects to be > >retired but not for the components. Thus, that might be not an option. > > > > Personally, I'm for the first approach. Does it sound reasonable? > > > > - > > Denis > > > > > > On Tue, Jun 18, 2019 at 7:39 AM Maxim Muzafarov > > wrote: > > > > > Folks, > > > > > > +1 to reduce the number of supported features. > > > > > > Probably, the best solution will be removing IGFS from core module and > > > making it as an Ignite plugin (will require some efforts to do this). > > > I've also think we can move IGFS to the separate branch (from the > > > master one) if someone will decide merge to latest changes from the > > > master branch to build Ignite from scratch with IGFS feature. > > > > > > On Mon, 17 Jun 2019 at 22:42, Denis Magda wrote: > > > > > > > > > > > > > > +1 from me to reduce supported feature list. > > > > > Guys, are we talking about Ignite 3.0? > > > > > > > > > > > > Nikolay, I would discontinue IGFS before 3.0. Let's do this in the > next > > > > release? As for other features and integrations, 3.0 should be > > considered > > > > as a version. > > > > > > > > > > > > > +1 from me provided that we move sources to some supplementary > > > repository. > > > > > If someone later would like to maintain/fix this module, he/she > > should > > > be > > > > > able to access sources with current state of the master. > > > > > > > > > > > > Dmitry, are you suggesting to move the sources to Github and abandon > > them > > > > there? Sort of legacy code cemetery. > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Mon, Jun 17, 2019 at 2:00 AM Nikolay Izhikov > > > > wrote: > > > > > > > > > +1 from me to reduce supported feature list. > > > > > > > > > > Guys, are we talking about Ignite 3.0? > > > > > > > > > > > > > > > В Пн, 17/06/2019 в 11:57 +0300, Alexey Goncharuk пишет: > > > > > > Denis, > > > > > > > > > > > > I fully support this idea. > > > > > > > > > > > > First, looking back, I do not think it was a good design in the > > first > > > > > place > > > > > > to build IGFS on top of Ignite caches. Second, I have never seen > a > > > case > > > > > > where IGFS provided significant performance boost. Usually it's > > > either > > > > > all > > > > > > data already fits buffer cache, and IGFS caching is not needed; > or > > > data > > > > > > does not fit buffer cache, and access pattern is close to full > scan > > > and > > > > > > additional caching in IGFS does not make sense. > > > > > > > > > > > > пн, 17 июн. 2019 г. в 11:28, Павлухин Иван >: > > > > > > > > > > > > > Denis, > > > > > > > > > > > > > > I must say that aforementioned solutions for a Hadoop ecosystem > > > appear > > > > > > > from time to time in questions on a user mailing list. So, it > > seems > > > > > > > that there is a practical need for such solutions. > > > > > > > > > > > > > > But of course it does not mean that we should continue a > support > > of > > > > > > > IGFS and Hadoop Accelerator. If both are not solutions that fit > > > well > > > > > > > common use cases then we should discontinue it. If any of them > is > > > very > > > > > > > good for it's purpo