[ANNOUNCE] Apache Ignite 3.0.0-alpha2 is released!

[Valentin Kulichenko]

Igniters,

I'm happy to announce that Ignite 3 project reached a significant
milestone, as we release the 2nd alpha version of the product.

On top of the functionality that was previously released, Alpha 2 adds
the following major features:
- Replication infrastructure based on Raft.
- New in-memory atomic storage with the basic insert-read functionality.
- New schema management engine and API.

There is an ability to create a cluster, and use the new Table to API to
write and read the data. Basic code examples are available here:
https://github.com/apache/ignite-3/tree/3.0.0-alpha2/examples

The best way to try the release out is to go through the Getting Started
Guide: https://ignite.apache.org/docs/3.0.0-alpha

If there are any questions, issues, or thoughts, please do not hesitate to
reply to this email. Ignite Community is welcoming any feedback and will
consider it in future development.

-Val

Re: How to add a support for encryption/decryption of third party persistence password

[Stephen Darlington]

The quick fix is you probably want to reference spring4 rather than spring3.


   
   


I tested it and it works for me, with Ignite 2.10.

However… if you’re willing to put the encryption password in an environment 
variable, why not just put the third-party DB password in the environment 
variable? I’m not convinced that the extra level of indirection adds much in 
the way of security. My preference is always to use a secrets manager like 
Vault or AWS Secret Manager.

> On 30 Jun 2021, at 06:23, Kamlesh Joshi  wrote:
> 
> Hi Igniters,
>  
> We want to add support for DB password encryption/decryption. We have tried 
> following approach but no luck (ignite is unable to decrypt the password). 
> Can anyone guide on this and if it’s a correct approach? Added supported jars 
> on Ignite classpath.
>  
> Ignite configuration file:
>  
>  class="org.jasypt.encryption.pbe.config.EnvironmentStringPBEConfig">
> 
> 
>
> 
>  class="org.jasypt.encryption.pbe.StandardPBEStringEncryptor">
>  ref="environmentVariablesConfiguration" />
> 
> 
>
> class="org.jasypt.spring3.properties.EncryptablePropertyPlaceholderConfigurer">
> 
> 
> 
>  
> Property file entry:
> thirdPartyDS.jdbc.password=ENC(AQqqmNSRdDQDic+9nMnFRq5rrkQZn)
>  
>  
> Thanks and Regards,
> Kamlesh Joshi

Re: Defrag?

[Ryan Trollip]

Hey Ilya

It's the data tables that keep growing not the WAL.
We will try to rebuild the cache and see if that fixes the issue

On Mon, Jun 28, 2021 at 8:46 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter, any
> specific caches that are growing unbound?
>
> If letter, you can try creating a new cache, moving the relevant data to
> this new cache, switch to using it, and then drop the old cache - should
> reclaim the space.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 28 июн. 2021 г. в 17:34, Ryan Trollip :
>
>> Is this why the native disk storage just keeps growing and does not
>> reduce after we delete from ignite using SQL?
>> We are up to 80GB on disk now on some instances. We implemented a custom
>> archiving feature to move older data out of ignite cache to a PostgresSQL
>> database but when we delete that data from ignite instance, the disk data
>> size ignite is using stays the same, and then keeps growing, and
>> growing
>>
>> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda  wrote:
>>
>>> Ignite fellows,
>>>
>>> I remember some of us worked on the persistence defragmentation
>>> features. Has it been merged?
>>>
>>> @Valentin Kulichenko  probably you know
>>> the latest state.
>>>
>>> -
>>> Denis
>>>
>>> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
 Hello!

 You can probably drop the entire cache and then re-populate it via
 loadCache(), etc.

 Regards,
 --
 Ilya Kasnacheev


 ср, 23 июн. 2021 г. в 21:47, Ryan Trollip :

> Thanks, Ilya, we may have to consider moving back to non-native
> storage and caching more selectively as the performance degrades when 
> there
> is a lot of write/delete activity or tables with large amounts of rows.
> This is with SQL with indexes and the use of query plans etc.
>
> Is there any easy way to rebuild the entire native database after
> hours? e.g. with a batch run on the weeknds?
>
> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com> wrote:
>
>> Hello!
>>
>> I don't think there's anything ready to use, but "killing
>> performance" from fragmentation is also not something reported too often.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip :
>>
>>> We see continual very large growth to data with ignite native. We
>>> have a very chatty use case that's creating and deleting stuff often. 
>>> The
>>> data on disk just keeps growing at an explosive rate. So much so we 
>>> ported
>>> this to a DB to see the difference and the DB is much smaller. I was
>>> searching to see if someone has the same issue. This is also killing
>>> performance.
>>>
>>> Founds this:
>>>
>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>
>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
>>> pages?
>>>
>>> Has anyone implemented a workaround to rebuild the cache and indexes
>>> say on a weekly basis to get it to behave reasonably?
>>>
>>> Thanks
>>>
>>

Re: Howto fix object lock in cache?

[Thomas Kramer]

Hi,

thanks for the hint using the control.sh script. It looks like the
server node with .111 IP address is having some very old transactions:

Matching transactions:
TcpDiscoveryNode [id=78b6df9f-7b77-46ea-9aef-a1b61918b258,
addrs=[110.10.123.111], order=1, ver=2.8.1#20200521-sha1:86422096,
isClient=false, consistentId=3d51530f-0dec-46da-98cb-eb550c860777]
    Tx: [xid=93724cf8971--0d9e-4326--0001,
label=null, state=MARKED_ROLLBACK, startTime=2021-05-24 06:22:10.778,
duration=3224128, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC,
topVer=AffinityTopologyVersion [topVer=74, minorTopVer=0], timeout=9990,
size=0, dhtNodes=[],
nearXid=f0724cf8971--0d9e-4326--004a,
parentNodeIds=[75431074]]
    Tx: [xid=7219dcbf971--0d9e-452c--0001,
label=null, state=MARKED_ROLLBACK, startTime=2021-06-23 13:12:53.094,
duration=607486, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC,
topVer=AffinityTopologyVersion [topVer=592, minorTopVer=0],
timeout=1, size=0, dhtNodes=[],
nearXid=f119dcbf971--0d9e-452c--01d9,
parentNodeIds=[458bca81]]
    Tx: [xid=a1480fb5a71--0d9e-4593--0001,
label=null, state=PREPARING, startTime=2021-06-30 10:51:37.874,
duration=11161, isolation=READ_COMMITTED, concurrency=OPTIMISTIC,
topVer=AffinityTopologyVersion [topVer=695, minorTopVer=0], timeout=0,
size=1, dhtNodes=[78b6df9f],
nearXid=a1480fb5a71--0d9e-4593--0001,
parentNodeIds=[78b6df9f]]
TcpDiscoveryNode [id=5eb48408-a239-4a92-9fb2-7903407975ad,
addrs=[110.10.123.87], order=688, ver=2.8.1#20200521-sha1:86422096,
isClient=true, consistentId=5eb48408-a239-4a92-9fb2-7903407975ad]
    Tx: [xid=e95d3fb5a71--0d9e-4593--02b0,
label=null, state=PREPARING, startTime=2021-06-30 13:57:39.511,
duration=0, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC,
topVer=AffinityTopologyVersion [topVer=695, minorTopVer=0], timeout=0,
size=1, dhtNodes=[78b6df9f],
nearXid=e95d3fb5a71--0d9e-4593--02b0,
parentNodeIds=[5eb48408]]
Command [TX] finished with code: 0

What happens if I would kill these two transactions that are in
MARKED_ROLLBACK state but look like they never finished? Is it safe to
kill them?

Thanks!


On 30.06.21 15:20, Zhenya Stanilovsky wrote:


Hi, first of all you need to know who is holding a lock, check [1]
command.
[1]
https://ignite.apache.org/docs/latest/tools/control-script#transaction-management


Hi all,

I have the situation that at least one DB row of my persistent cache
seems locked and I can't change it anymore. Everytime I want to change
it using SQL a TransactionTimeoutException happens like this:

class org.apache.ignite.transactions.TransactionTimeoutException:
Failed
to acquire lock within provided timeout for transaction
[timeout=1,
tx=GridNearTxLocal[xid=cc2d4db5a71--0d9e-4593--02b6,
xidVersion=GridCacheVersion [topVer=228476307, order=1625038312140,
nodeOrder=694], nearXidVersion=GridCacheVersion [topVer=228476307,
order=1625038312140, nodeOrder=694], concurrency=PESSIMISTIC,
isolation=REPEATABLE_READ, state=MARKED_ROLLBACK, invalidate=false,
rollbackOnly=true, nodeId=62e91173-e912-49d5-b238-e95f8fe38314,
timeout=1, startTime=1625051540461, duration=10034, label=null]]

However, the system continues to run and other objects in the
cache can
be added and changed.

Is there a way to unlock this particular entry?

Thanks!

Re: Howto fix object lock in cache?

[Zhenya Stanilovsky]

Hi, first of all you need to know who is holding a lock, check [1] command.
 
[1]  
https://ignite.apache.org/docs/latest/tools/control-script#transaction-management
 
>Hi all,
>
>I have the situation that at least one DB row of my persistent cache
>seems locked and I can't change it anymore. Everytime I want to change
>it using SQL a TransactionTimeoutException happens like this:
>
>class org.apache.ignite.transactions.TransactionTimeoutException: Failed
>to acquire lock within provided timeout for transaction [timeout=1,
>tx=GridNearTxLocal[xid=cc2d4db5a71--0d9e-4593--02b6,
>xidVersion=GridCacheVersion [topVer=228476307, order=1625038312140,
>nodeOrder=694], nearXidVersion=GridCacheVersion [topVer=228476307,
>order=1625038312140, nodeOrder=694], concurrency=PESSIMISTIC,
>isolation=REPEATABLE_READ, state=MARKED_ROLLBACK, invalidate=false,
>rollbackOnly=true, nodeId=62e91173-e912-49d5-b238-e95f8fe38314,
>timeout=1, startTime=1625051540461, duration=10034, label=null]]
>
>However, the system continues to run and other objects in the cache can
>be added and changed.
>
>Is there a way to unlock this particular entry?
>
>Thanks!
>  
 
 
 
 

Howto fix object lock in cache?

[Thomas Kramer]

Hi all,

I have the situation that at least one DB row of my persistent cache
seems locked and I can't change it anymore. Everytime I want to change
it using SQL a TransactionTimeoutException happens like this:

class org.apache.ignite.transactions.TransactionTimeoutException: Failed
to acquire lock within provided timeout for transaction [timeout=1,
tx=GridNearTxLocal[xid=cc2d4db5a71--0d9e-4593--02b6,
xidVersion=GridCacheVersion [topVer=228476307, order=1625038312140,
nodeOrder=694], nearXidVersion=GridCacheVersion [topVer=228476307,
order=1625038312140, nodeOrder=694], concurrency=PESSIMISTIC,
isolation=REPEATABLE_READ, state=MARKED_ROLLBACK, invalidate=false,
rollbackOnly=true, nodeId=62e91173-e912-49d5-b238-e95f8fe38314,
timeout=1, startTime=1625051540461, duration=10034, label=null]]

However, the system continues to run and other objects in the cache can
be added and changed.

Is there a way to unlock this particular entry?

Thanks!