All of this looks awesome, covers the use cases I know about.
Thanks!
Stan
> On 10 Aug 2020, at 15:39, ткаленко кирилл <tkalkir...@yandex.ru> wrote:
>
> Hi, Stan again :-)
>
> I suggest adding a little more flexibility to configuration:
> 1)Add default warm-up configuration for all regions into
> org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration
> 2)Add a NoOp strategy for turning off heating of a specific region
>
> Thus, when starting warm-up, region configuration is taken at beginning, and
> if it is not present, it is taken from default. And if we don't want to warm
> up region, we set NoOp.
>
> 10.08.2020, 10:20, "ткаленко кирилл" <tkalkir...@yandex.ru>:
>> Hi, Stan!
>>
>> As a result of personal correspondence I realized that you are right about
>> making changes:
>> 1)Move warm-up configuration to
>> org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
>> 2)Start warming up for each region sequentially;
>> 3)Improving warm-up interface:
>>
>> package org.apache.ignite.internal.processors.cache.warmup;
>>
>> import org.apache.ignite.IgniteCheckedException;
>> import org.apache.ignite.configuration.WarmUpConfiguration;
>> import org.apache.ignite.internal.GridKernalContext;
>> import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
>>
>> /**
>> * Interface for warming up.
>> */
>> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>> /**
>> * Returns configuration class for mapping to strategy.
>> *
>> * @return Configuration class.
>> */
>> Class<T> configClass();
>>
>> /**
>> * Warm up.
>> *
>> * @param kernalCtx Kernal context.
>> * @param cfg Warm-up configuration.
>> * @param region Data region.
>> * @throws IgniteCheckedException if faild.
>> */
>> void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region)
>> throws IgniteCheckedException;
>>
>> /**
>> * Closing warm up.
>> *
>> * @throws IgniteCheckedException if faild.
>> */
>> void close() throws IgniteCheckedException;
>> }
>>
>> 4)Add a command to "control.sh", to stop current warm-up and cancel all
>> others: --warm-up stop
>> 5)The "load all" strategy will work as long as there is enough RAM and index
>> pages will also take priority.
>>
>> 04.08.2020, 13:29, "ткаленко кирилл" <tkalkir...@yandex.ru>:
>>> Hi, Slava!
>>>
>>> Thank you for looking at the offer and making fair comments.
>>>
>>> I personally discussed with Anton and Alexey because they are author and
>>> sponsor of "IEP-40" and we found out that point 2 in it is no longer
>>> relevant and it can be removed.
>>> I suggest implementing point 3, since it may be independent of point 1.
>>> Also, the warm-up will always start after restore phase, without
>>> subscribing to events.
>>>
>>> You are right this should be mentioned in the documentation and javadoc.
>>>> This means that the user's thread, which starts Ignite via
>>>> Ignition.start(), will wait for ana additional step - cache warm-up.
>>>> I think this fact has to be clearly mentioned in our documentation (at
>>>> Javadocat least) because this step can be time-consuming.
>>>
>>> My suggestion for implementation:
>>> 1)Adding a marker interface
>>> "org.apache.ignite.configuration.WarmUpConfiguration" for configuring cache
>>> warming;
>>> 2)Set only one configuration via
>>> "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
>>> 3)Add an internal warm-up interface that will start in [1] after [2];
>>>
>>> package org.apache.ignite.internal.processors.cache.warmup;
>>>
>>> import org.apache.ignite.IgniteCheckedException;
>>> import org.apache.ignite.configuration.WarmUpConfiguration;
>>> import org.apache.ignite.internal.GridKernalContext;
>>>
>>> /**
>>> * Interface for warming up.
>>> */
>>> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>>> /**
>>> * Returns configuration class for mapping to strategy.
>>> *
>>> * @return Configuration class.
>>> */
>>> Class<T> configClass();
>>>
>>> /**
>>> * Warm up.
>>> *
>>> * @param kernalCtx Kernal context.
>>> * @param cfg Warm-up configuration.
>>> * @throws IgniteCheckedException if faild.
>>> */
>>> void warmUp(GridKernalContext kernalCtx, T cfg) throws
>>> IgniteCheckedException;
>>> }
>>>
>>> 4)Adding an internal plugin extension for add own strategies;
>>>
>>> package org.apache.ignite.internal.processors.cache.warmup;
>>>
>>> import java.util.Collection;
>>> import org.apache.ignite.plugin.Extension;
>>>
>>> /**
>>> * Interface for getting warm-up strategies from plugins.
>>> */
>>> public interface WarmUpStrategySupplier extends Extension {
>>> /**
>>> * Getting warm-up strategies.
>>> *
>>> * @return Warm-up strategies.
>>> */
>>> Collection<WarmUpStrategy> strategies();
>>> }
>>>
>>> 5)Add a "Load all" strategy that will load everything to memory as long as
>>> there is space in it. This strategy is suitable if the persistent storage
>>> is less than RAM.
>>>
>>> Any objections or comments?
>>>
>>> [1] -
>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>> [2] -
>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>
>>> 27.07.2020, 16:48, "Вячеслав Коптилин" <slava.kopti...@gmail.com>:
>>>> Hello Kirill,
>>>>
>>>> Thanks a lot for driving this activity. If I am not mistaken, this
>>>> discussion relates to IEP-40.
>>>>
>>>>> I suggest adding a warmup phase after recovery here [1] after [2],
>>>>> before
>>>>
>>>> discovery.
>>>> This means that the user's thread, which starts Ignite via
>>>> Ignition.start(), will wait for ana additional step - cache warm-up.
>>>> I think this fact has to be clearly mentioned in our documentation (at
>>>> Javadocat least) because this step can be time-consuming.
>>>>
>>>>> I suggest adding a new interface:
>>>>
>>>> I would change it a bit. First of all, it would be nice to place this
>>>> interface to a public package and get rid of using GridCacheContext,
>>>> which is an internal class and it should not leak to the public API in
>>>> any
>>>> case.
>>>> Perhaps, this parameter is not needed at all or we should add some public
>>>> abstraction instead of internal class.
>>>>
>>>> package org.apache.ignite.configuration;
>>>>
>>>> import org.apache.ignite.IgniteCheckedException;
>>>> import org.apache.ignite.lang.IgniteFuture;
>>>>
>>>> public interface CacheWarmupper {
>>>> /**
>>>> * Warmup cache.
>>>> *
>>>> * @param cachename Cache name.
>>>> * @return Future cache warmup.
>>>> * @throws IgniteCheckedException If failed.
>>>> */
>>>> IgniteFuture<?> warmup(String cachename) throws
>>>> IgniteCheckedException;
>>>> }
>>>>
>>>> Thanks,
>>>> S.
>>>>
>>>> пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл <tkalkir...@yandex.ru>:
>>>>
>>>>> Now, after restarting node, we have only cold caches, which at first
>>>>> requests to them will gradually load data from disks, which can slow
>>>>> down
>>>>> first calls to them.
>>>>> If node has more RAM than data on disk, then they can be loaded at
>>>>> start
>>>>> "warmup", thereby solving the issue of slowdowns during first calls to
>>>>> caches.
>>>>>
>>>>> I suggest adding a warmup phase after recovery here [1] after [2],
>>>>> before
>>>>> descovery.
>>>>>
>>>>> I suggest adding a new interface:
>>>>>
>>>>> package org.apache.ignite.internal.processors.cache;
>>>>>
>>>>> import org.apache.ignite.IgniteCheckedException;
>>>>> import org.apache.ignite.internal.IgniteInternalFuture;
>>>>> import org.jetbrains.annotations.Nullable;
>>>>>
>>>>> /**
>>>>> * Interface for warming up cache.
>>>>> */
>>>>> public interface CacheWarmup {
>>>>> /**
>>>>> * Warmup cache.
>>>>> *
>>>>> * @param cacheCtx Cache context.
>>>>> * @return Future cache warmup.
>>>>> * @throws IgniteCheckedException if failed.
>>>>> */
>>>>> @Nullable IgniteInternalFuture<?> process(GridCacheContext
>>>>> cacheCtx)
>>>>> throws IgniteCheckedException;
>>>>> }
>>>>>
>>>>> Which will allow to warm up caches in parallel and asynchronously.
>>>>> Warmup
>>>>> phase will end after all IgniteInternalFuture for all caches isDone.
>>>>>
>>>>> Also adding the ability to customize via methods:
>>>>>
>>>>> org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>>>> org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>>>>
>>>>> Which will allow for each cache to set implementation of cache warming
>>>>> up,
>>>>> both for a specific cache, and for all if necessary.
>>>>>
>>>>> I suggest adding an implementation of SequentialWarmup that will use
>>>>> [3].
>>>>>
>>>>> Questions, suggestions, comments?
>>>>>
>>>>> [1] -
>>>>>
>>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>>> [2] -
>>>>>
>>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>>> [3] -
>>>>>
>>>>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload
