RE: Adopting mock framework (ex. Mockito) for unit tests

[Jason Man, CLSA]

As discussed, updated the PR with the change to put the mockito version at the 
parent POM and using it on the cassandra module.

https://github.com/apache/ignite/pull/3088

Thanks.

-Original Message-
From: Denis Magda [mailto:dma...@apache.org] 
Sent: Wednesday, January 17, 2018 9:55 AM
To: dev@ignite.apache.org; Igor Rudyak
Subject: Re: Adopting mock framework (ex. Mockito) for unit tests

Looks we came to a consensus. Jason, please change the PR the way you say

> I will change my PR to add the dependency at parent POM level and use it only 
> for my test in ignite-cassandra.

Igor should review it shortly.

—
Denis

> On Jan 16, 2018, at 5:33 PM, Jason Man, CLSA  wrote:
> 
> Agree with Vladimir/Denis/Igor on using it for specific tests/components.
> 
> What I meant by 'adopt' is to be able use it to facilitate testing of Ignite 
> code somewhere.  It doesn’t mean we should change our overall testing 
> approach overnight and try to use it everywhere.
> 
> In the case of testing code that integrates 3rd party tools (Postgres, MySQL, 
> Cassandra), I think it definitely simplifies the test development effort as 
> well as the time it takes to run the tests.
> 
> @Alexey, I haven't explored using JMockit, but Mockito was just a very 
> popular mocking framework and widely used so it would be less of a learning 
> curve for most developers:
> https://trends.google.com/trends/explore?q=jmockit,mockito
> 
> I will change my PR to add the dependency at parent POM level and use it only 
> for my test in ignite-cassandra.
> 
> Jason
> 
> -Original Message-
> From: Denis Magda [mailto:dma...@apache.org]
> Sent: Wednesday, January 17, 2018 3:53 AM
> To: dev@ignite.apache.org
> Subject: Re: Adopting mock framework (ex. Mockito) for unit tests
> 
> Agree with Igor’s opinion. If it simplifies Cassandra integration testing 
> cycles and maintenance then we should go for it for *this* component only. No 
> need to push it to all the component we have.
> 
> —
> Denis
> 
>> On Jan 16, 2018, at 10:24 AM, Igor Rudyak  wrote:
>> 
>> It will be good to clarify what do you mean by adopt? Can't we just 
>> start using it as is for specific cases?
>> 
>> I understand that there are some cases which probably not the best 
>> scenario for Mockito. At the same time there are lot's of other cases 
>> (like in
>> IGNITE-6853 <https://issues.apache.org/jira/browse/IGNITE-6853>) when 
>> tests could be significantly simplified using mock framework.
>> 
>> May be it makes sense to introduce mock framework at the parent POM 
>> and then just reuse it at specific modules for the test cases where 
>> appropriate?
>> 
>> Igor
>> 
>> On Tue, Jan 16, 2018 at 4:27 AM, Vladimir Ozerov 
>> 
>> wrote:
>> 
>>> Mocking is a good testing technique, but over years we failed to 
>>> adopt it in Ignite. The reason is high project complexity, when a 
>>> lot of components are interact with each other, and it is very hard 
>>> to extract clean interfaces out of it. We definitely could do better 
>>> with our OOP, but you should remember that good OOP comes at costs - 
>>> more code, more time, worse performance (due to lot's of various 
>>> wrappers). I think of it as a normal case based on my experience - I 
>>> worked with a lot of code bases (Postgres, MySQL, Cassandra, 
>>> Hazelcast, to name a few) - and none of them are clean enough to 
>>> adopt mocking easily. You hardly find clean code in performance-demanded 
>>> projects.
>>> 
>>> TDD is also controversial approach for complex projects. It works 
>>> good when you work on concrete specification of the task and know 
>>> the outcome in advance. But it doesn't work for Ignite - typically 
>>> we do not know outcomes of our activities in advance. Things could 
>>> change dramatically during developments, so TDD for us is waste of time.
>>> 
>>> On the other hand, today we are able to "mock" a lot of internal 
>>> components by hands to test various complex cases. E.g. one can 
>>> easily add his own IO manager to test message drops. You can do virtually 
>>> everything you need.
>>> 
>>> For this reason I doubt mocking is the right approach we should 
>>> think of for core development. But it may do great job for 
>>> integration with 3rd-party products.
>>> 
>>> Vladimir.
>>> 
>>> On Tue, Jan 16, 2018 at 1:34 PM, Alexey Kukushkin < 
>>> kukushkinale...@gmail.com
>>>> wrote:
>>> 
>>>&g

RE: Adopting mock framework (ex. Mockito) for unit tests

[Jason Man, CLSA]

Agree with Vladimir/Denis/Igor on using it for specific tests/components.

What I meant by 'adopt' is to be able use it to facilitate testing of Ignite 
code somewhere.  It doesn’t mean we should change our overall testing approach 
overnight and try to use it everywhere.

In the case of testing code that integrates 3rd party tools (Postgres, MySQL, 
Cassandra), I think it definitely simplifies the test development effort as 
well as the time it takes to run the tests.

@Alexey, I haven't explored using JMockit, but Mockito was just a very popular 
mocking framework and widely used so it would be less of a learning curve for 
most developers:
https://trends.google.com/trends/explore?q=jmockit,mockito

I will change my PR to add the dependency at parent POM level and use it only 
for my test in ignite-cassandra.

Jason

-Original Message-
From: Denis Magda [mailto:dma...@apache.org] 
Sent: Wednesday, January 17, 2018 3:53 AM
To: dev@ignite.apache.org
Subject: Re: Adopting mock framework (ex. Mockito) for unit tests

Agree with Igor’s opinion. If it simplifies Cassandra integration testing 
cycles and maintenance then we should go for it for *this* component only. No 
need to push it to all the component we have.

—
Denis

> On Jan 16, 2018, at 10:24 AM, Igor Rudyak  wrote:
> 
> It will be good to clarify what do you mean by adopt? Can't we just 
> start using it as is for specific cases?
> 
> I understand that there are some cases which probably not the best 
> scenario for Mockito. At the same time there are lot's of other cases 
> (like in
> IGNITE-6853 ) when 
> tests could be significantly simplified using mock framework.
> 
> May be it makes sense to introduce mock framework at the parent POM 
> and then just reuse it at specific modules for the test cases where 
> appropriate?
> 
> Igor
> 
> On Tue, Jan 16, 2018 at 4:27 AM, Vladimir Ozerov 
> 
> wrote:
> 
>> Mocking is a good testing technique, but over years we failed to 
>> adopt it in Ignite. The reason is high project complexity, when a lot 
>> of components are interact with each other, and it is very hard to 
>> extract clean interfaces out of it. We definitely could do better 
>> with our OOP, but you should remember that good OOP comes at costs - 
>> more code, more time, worse performance (due to lot's of various 
>> wrappers). I think of it as a normal case based on my experience - I 
>> worked with a lot of code bases (Postgres, MySQL, Cassandra, 
>> Hazelcast, to name a few) - and none of them are clean enough to 
>> adopt mocking easily. You hardly find clean code in performance-demanded 
>> projects.
>> 
>> TDD is also controversial approach for complex projects. It works 
>> good when you work on concrete specification of the task and know the 
>> outcome in advance. But it doesn't work for Ignite - typically we do 
>> not know outcomes of our activities in advance. Things could change 
>> dramatically during developments, so TDD for us is waste of time.
>> 
>> On the other hand, today we are able to "mock" a lot of internal 
>> components by hands to test various complex cases. E.g. one can 
>> easily add his own IO manager to test message drops. You can do virtually 
>> everything you need.
>> 
>> For this reason I doubt mocking is the right approach we should think 
>> of for core development. But it may do great job for integration with 
>> 3rd-party products.
>> 
>> Vladimir.
>> 
>> On Tue, Jan 16, 2018 at 1:34 PM, Alexey Kukushkin < 
>> kukushkinale...@gmail.com
>>> wrote:
>> 
>>> Hi Jason,
>>> 
>>> I heartily support unit testing practices and introducing a mocking 
>>> framework into ignite development environment. Today I can hardly 
>>> find a single unit test in Apache Ignite, which does not allow me to 
>>> use the
>> best
>>> TDD and CI/CD practices like running tests on every commit (or even 
>>> on every "save file"!).
>>> 
>>> I recently started developing an isolated component based on Apache
>> Ignite
>>> and, since I use TDD and write lots of unit tests, I had to add a 
>>> mocking framework to my project. I started from Mockito (version 
>>> 1.something) and found I could not do some things with Mockito due 
>>> to Ignite currently not designed for unit testing. I believe I could 
>>> not find a way to mock some private initialisation block with Mockito.
>>> 
>>> Thus, I switched to JMockit - it allowed me to mock what I wanted 
>>> and it seems you can mock virtually everything with JMockit.
>>> 
>>> I know that a situation when you have to mock something private or 
>>> static indicates not very modular and extendable design but you do 
>>> not have much of a choice with Ignite since you already have huge 
>>> amount of code and it would be really hard to refactor everything to 
>>> make it testable since Ignite development process is heavy and your 
>>> project could be stuck
>> waiting
>>> for Ignite changes.
>>> 
>>> Did you consider JMockit over Mockito

Adopting mock framework (ex. Mockito) for unit tests

[Jason Man, CLSA]

Hi Igniters,

I'd like to discuss the topic of adopting a mock framework such as Mockito into 
Ignite's project for writing unit tests.

Here's a simple article that illustrate why using a mock object for testing is 
necessary, and why using a framework is better than doing it yourself:
https://zeroturnaround.com/rebellabs/how-to-mock-up-your-unit-test-environment-to-create-alternate-realities/

Here's Mokito's website which also illustrate the Why and How:
http://site.mockito.org/

Then, here's my PR for one of the bug fixes I'm trying to fix in Ignite that 
prompted me to use mocks:
https://github.com/apache/ignite/pull/3088/files

In the case of my unit test, not using a mocking framework will be difficult to 
test certain negative behaviour of Cassandra.  On the other hand, manually 
mocking the objects (instead of using a framework) will result in hundreds of 
lines of boiler plate code.  Or in some cases, it's ugly to mock an object 
where it has a package-private constructor instead of a public one. (you'd have 
to put your mock object in the same package...)

Overall, I think introducing a mocking framework can help us write unit tests 
that have better coverage of code, especially mocking behaviour of external 
dependencies (such as Cassandra).

What do you guys think?

Regards,
Jason

The content of this communication is intended for the recipient and is subject 
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon 
request.
Please consider before printing. CLSA is ISO14001 certified and committed to 
reducing its impact on the environment.

RE: IGNITE-6853: Cassandra cache store does not clean prepared statements cache when remove old cassandra session

[Jason Man, CLSA]

Thanks.  Just replied on the jira thread as well.  Thanks.

-Original Message-
From: Denis Magda [mailto:dma...@apache.org] 
Sent: Saturday, January 13, 2018 1:41 AM
To: Igor Rudyak
Cc: Jason Man, CLSA; dev@ignite.apache.org
Subject: Re: IGNITE-6853: Cassandra cache store does not clean prepared 
statements cache when remove old cassandra session

Just for the records.

Igor replied and reviewed the changes:
https://issues.apache.org/jira/browse/IGNITE-6853 
<https://issues.apache.org/jira/browse/IGNITE-6853>

—
Denis

> On Jan 11, 2018, at 9:13 AM, Igor Rudyak  wrote:
> 
> Hi Denis, Jason
> 
> Sorry just found your email regarding IGNITE-6853. It's being a long 
> Christmas holiday and your email got lost among tons of other emails. 
> 
> Anyway, I'll look at this PR: 
> https://github.com/apache/ignite/pull/3088/files 
> <https://github.com/apache/ignite/pull/3088/files>
> 
> Igor
>  
> 
> On Wed, Dec 20, 2017 at 10:15 PM, Denis Magda  <mailto:dma...@apache.org>> wrote:
> Hi Igor,
> 
> Could you please do the review by the end of the year? It will be great to 
> include the fix into 2.4.
> 
> —
> Denis
> 
>> Begin forwarded message:
>> 
>> From: Denis Magda mailto:dma...@apache.org>>
>> Subject: Re: IGNITE-6853: Cassandra cache store does not clean prepared 
>> statements cache when remove old cassandra session
>> Date: December 20, 2017 at 10:14:33 PM PST
>> To: dev@ignite.apache.org <mailto:dev@ignite.apache.org>, Igor Rudyak 
>> mailto:irud...@gmail.com>>
>> 
>> Hi Jason,
>> 
>> Thanks for fixing the bug! Please assign the ticket on yourself, move it to 
>> “PATCH AVAILABLE” state and run Cassandra specific tests on TeamCity. More 
>> on this can be found here:
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-SubmittingforReview
>>  
>> <https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-SubmittingforReview>
>> 
>> Igor Rudyak (copied) is a main maintainer. Igor, would you mind checking the 
>> patch and explain how to test the integration on TeamCity (if needed)?
>> 
>> —
>> Denis
>> 
>>> On Dec 20, 2017, at 10:08 PM, Jason Man, CLSA >> <mailto:jason@clsa.com>> wrote:
>>> 
>>> Hi Igniters,
>>> 
>>> I've got a PR to resolve this JIRA available:
>>> 
>>> https://issues.apache.org/jira/browse/IGNITE-6853 
>>> <https://issues.apache.org/jira/browse/IGNITE-6853>
>>> https://github.com/apache/ignite/pull/3088 
>>> <https://github.com/apache/ignite/pull/3088>
>>> 
>>> The JIRA was tagged previously with fix version = 2.4
>>> 
>>> May I know what is the process to move things forward?  (The PR has been 
>>> created 3 weeks ago).  I'm hoping this fix would be included in 2.4 
>>> release. 
>>> 
>>> Thanks.
>>> Jason  
>>> The content of this communication is intended for the recipient and is 
>>> subject to CLSA Legal and Regulatory Notices.
>>> These can be viewed at https://www.clsa.com/disclaimer.html 
>>> <https://www.clsa.com/disclaimer.html> or sent to you upon request.
>>> CLSA is ISO14001 certified and committed to reducing environmental impact.
>> 
> 
> 

The content of this communication is intended for the recipient and is subject 
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon 
request.
Please consider before printing. CLSA is ISO14001 certified and committed to 
reducing its impact on the environment.

RE: IGNITE-6853: Cassandra cache store does not clean prepared statements cache when remove old cassandra session

[Jason Man, CLSA]

Hi Ignor,

Would you please help review my patch to see if it is ok.  I'm hoping to 
include this in 2.4.

Thanks
Jason

-Original Message-
From: Denis Magda [mailto:dma...@apache.org] 
Sent: Thursday, December 21, 2017 2:15 PM
To: dev@ignite.apache.org; Igor Rudyak
Subject: Re: IGNITE-6853: Cassandra cache store does not clean prepared 
statements cache when remove old cassandra session

Hi Jason,

Thanks for fixing the bug! Please assign the ticket on yourself, move it to 
“PATCH AVAILABLE” state and run Cassandra specific tests on TeamCity. More on 
this can be found here:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-SubmittingforReview
 
<https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-SubmittingforReview>

Igor Rudyak (copied) is a main maintainer. Igor, would you mind checking the 
patch and explain how to test the integration on TeamCity (if needed)?

—
Denis

> On Dec 20, 2017, at 10:08 PM, Jason Man, CLSA  wrote:
> 
> Hi Igniters,
> 
> I've got a PR to resolve this JIRA available:
> 
> https://issues.apache.org/jira/browse/IGNITE-6853
> https://github.com/apache/ignite/pull/3088
> 
> The JIRA was tagged previously with fix version = 2.4
> 
> May I know what is the process to move things forward?  (The PR has been 
> created 3 weeks ago).  I'm hoping this fix would be included in 2.4 release. 
> 
> Thanks.
> Jason  
> The content of this communication is intended for the recipient and is 
> subject to CLSA Legal and Regulatory Notices.
> These can be viewed at https://www.clsa.com/disclaimer.html or sent to you 
> upon request.
> CLSA is ISO14001 certified and committed to reducing environmental impact.

The content of this communication is intended for the recipient and is subject 
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon 
request.
Please consider before printing. CLSA is ISO14001 certified and committed to 
reducing its impact on the environment.

IGNITE-6853: Cassandra cache store does not clean prepared statements cache when remove old cassandra session

[Jason Man, CLSA]

Hi Igniters,

I've got a PR to resolve this JIRA available:

https://issues.apache.org/jira/browse/IGNITE-6853
https://github.com/apache/ignite/pull/3088

The JIRA was tagged previously with fix version = 2.4

May I know what is the process to move things forward?  (The PR has been 
created 3 weeks ago).  I'm hoping this fix would be included in 2.4 release. 

Thanks.
Jason  
The content of this communication is intended for the recipient and is subject 
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon 
request.
CLSA is ISO14001 certified and committed to reducing environmental impact.

Why isn't Apache Ignite on stackshare.io

[Jason Man, CLSA]

Just curious, why isn't Apache Ignite on Stackshare.io?  Stackshare.io is 
really picking up in momentum as a developer community for sharing what stacks 
are used and where.  Could be useful in promoting usage in the open-source 
community.

I can see that Hazelcast is:
https://stackshare.io/hazelcast

Regards,
Jason

The content of this communication is intended for the recipient and is subject 
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon 
request.
CLSA is ISO14001 certified and committed to reducing environmental impact.