Re: KafkaStreamer, how to manage (stop consumming, resume) on client disconnection

[akorensh]

flush() guarantees completion of all futures returned by addData(Object,
Object)
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html#flush--

flush() will send the batch, but it is still possible for the server to
crash before the message reaches it.

If you need verify whether the particular put actually made it to the
server, appropriate events are available.
use these events:
https://ignite.apache.org/docs/latest/events/events#cache-events

You can put in retry logic if the client has disconnected.

One possibility is to use the CacheException as per the contract of the
flush() method 
you will get something like this: 
javax.cache.CacheException: class
org.apache.ignite.IgniteClientDisconnectedException: Data streamer has been
closed, client node disconnected.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1275)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.doFlush(DataStreamerImpl.java:1204)






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: KafkaStreamer, how to manage (stop consumming, resume) on client disconnection

[Pelado]

Thanks for the answers.
I resolved the problem of reconnection using events. It worked very well.
What I found, is the following...
The KafkaStreamer consumes records and send them to the IgniteDataStreamer.
It doesn't handle the IgniteFuture returned.
If the connection with the server is interrupted (server restart for
example) the KafkaStreamer is stoped, kafka consumers are stoped, but those
records that were sent to the streamer and (I believe) are in the buffer
are still trying to be saved in the cache.
There is no way to recover them as far as I know.
Am I right?
Should I implement a custom KafkaStreamer that, in that situation, handles
the IgniteFuture and let's say retry the insertion in the cache?

Another question, I'm using a grid service to start the streamer. What is
the benefit of this vs a simple spring service if I'm using kubernetes for
deployment?




On Fri, Nov 20, 2020 at 5:01 PM akorensh  wrote:

> Hi,
>   I think listening to events would be a good solution for you.
>
> There are two discovery events that are triggered on the client node when
> it
> is disconnected from or reconnected to the cluster:
>
> EVT_CLIENT_NODE_DISCONNECTED
>
> EVT_CLIENT_NODE_RECONNECTED
>
> see:
>
> https://ignite.apache.org/docs/latest/clustering/connect-client-nodes#client-disconnectedreconnected-events
>
>
> As for StreamReceiver: Keep in mind that the logic implemented in a stream
> receiver is executed on the node where data is to be stored.  If the server
> where the data resides crashes, your code might not execute.
> https://ignite.apache.org/docs/latest/data-streaming#stream-visitor
>
> Thanks, Alex
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


-- 
Facundo Maldonado

Re: KafkaStreamer, how to manage (stop consumming, resume) on client disconnection

[akorensh]

Hi,
  I think listening to events would be a good solution for you. 

There are two discovery events that are triggered on the client node when it
is disconnected from or reconnected to the cluster:

EVT_CLIENT_NODE_DISCONNECTED

EVT_CLIENT_NODE_RECONNECTED

see:
https://ignite.apache.org/docs/latest/clustering/connect-client-nodes#client-disconnectedreconnected-events


As for StreamReceiver: Keep in mind that the logic implemented in a stream
receiver is executed on the node where data is to be stored.  If the server
where the data resides crashes, your code might not execute.
https://ignite.apache.org/docs/latest/data-streaming#stream-visitor

Thanks, Alex





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

C# CacheStoreAdapter - Customizing Load, LoadCache methods

[ABDumalagan]

Hello, 
I have some confusion about implementing my own Load and LoadCache methods - 
I am trying to load data from an underlying Oracle database into cache, and
the entries are key-value pairs where the key is the Associate ID and the
value is the entire Associate object. 
*(1)* Regarding the Load method, I was under the impression that calling
Get(key) would inherently call the Load(key) method from my OracleStore.cs
file and load the key and corresponding object from the database into cache. 
Under the Load method in OracleStore.cs, I connect to the underlying Oracle
database and execute a query which grabs the data that correlates to the
Associate ID.
However, after running my Program.cs file, I received an error saying
*"System.Collections.Generic.KeyNotFoundException: 'The given key was not
present in the cache: 18755"* -- was I under the wrong impression that the
Load(key) would just put the data into cache? 
I know that the Load method returns an object, and in my case, it returns an
Associate object. However, do I need to do implement something else in order
to put the object returned from the Load method into cache?
*(2)* With the LoadCache method, I essentially would be doing the same thing
as the Load method, but this time, I would be querying 10,000 entries from
the database and putting it into cache. I also hold the same assumption that
calling the LoadCache method will just load the 10,000 entries in cache.
Also, I am unsure about what to do for the Action parameter in
LoadCache(Action, Object[])--would I need to define a method to pass
as a parameter, and would that method be responsible for putting the entries
in cache? If so, how would I go about it? 
 
Files attached below:
Inherits CacheStoreAdapter:  OracleStore.cs
  
Inherits IFactory:  OracleStoreFactory.cs

  
Establishes Oracle database connection:  DBConnection.cs
  
Defines Associate object:  Associate.cs
  
Runs program (creates and starts node):  Program.cs
  




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

IgniteSecurity vs GridSecurityProcessor

[Vishwas Bm]

Hi,

We were using 2.7.6 and had implemented a custom security plugin for
authorization and authentication by implementing GridSecurityProcessor.

Now in 2.9 we see that a new interface is provided IgniteSecurity.
May I know what is the difference between the interfaces, as both look
similar and what is appropriate place to implement them.

Also in 2.7.6 there was a class called SecurityContextHolder to  hold the
context.
Now in 2.9 we do not see that class and we see a class
OperartionClassContext.
How do we use this new class when using a custom security plugin?



Regards,
Vishwas

Re: [2.8.1]Checking optimistic transaction state on remote nodes

[Ilya Kasnacheev]

Hello!

It will happen when the node has left but the transaction has to be
committed.

Most operations on transactional cache will involve implicit transactions
so there you go.

Regards,
-- 
Ilya Kasnacheev


чт, 19 нояб. 2020 г. в 16:46, 38797715 <38797...@qq.com>:

> Hi community,
>
> Although there is a transactional cache, no transaction operation is
> performed, but there is a lot of output below in the log. Why?
>
> [2020-11-16 14:01:44,947][INFO ][sys-stripe-8-#9][IgniteTxManager]
> Checking optimistic transaction state on remote nodes [tx=GridDhtTxLocal
> [nearNodeId=a7eded9b-4078-4ee5-a1dd-426b8debc203,
> nearFutId=e0576afd571-dbd82c53-1772-4c53-a4ea-38e601002379,
> nearMiniId=1, nearFinFutId=null, nearFinMiniId=0,
> nearXidVer=GridCacheVersion [topVer=216485010, order=1607062821327,
> nodeOrder=30], lb=null, super=GridDhtTxLocalAdapter
> [nearOnOriginatingNode=false, nearNodes=KeySetView [],
> dhtNodes=KeySetView [e4d4fc27-d2d9-47f9-8d21-dfac2c003b55,
> 3060fc02-e94a-4b6d-851a-05d75ea751e0], explicitLock=false,
> super=IgniteTxLocalAdapter [completedBase=null,
> sndTransformedVals=false, depEnabled=false,
> txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false,
> useMvccCaching=false], super=IgniteTxAdapter [xidVer=GridCacheVersion
> [topVer=216485010, order=1607062856849, nodeOrder=1],
> writeVer=GridCacheVersion [topVer=216485023, order=1607062856850,
> nodeOrder=1], implicit=true, loc=true, threadId=24070,
> startTime=1605506134277, nodeId=2b0db4f4-86d1-42c2-babf-f6318bd932e5,
> startVer=GridCacheVersion [topVer=216485010, order=1607062856849,
> nodeOrder=1], endVer=null, isolation=READ_COMMITTED,
> concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false,
> plc=2, commitVer=null, finalizing=RECOVERY_FINISH, invalidParts=null,
> state=PREPARED, timedOut=false, topVer=AffinityTopologyVersion
> [topVer=117, minorTopVer=0], mvccSnapshot=null, skipCompletedVers=false,
> parentTx=null, duration=370668ms, onePhaseCommit=false], size=1]]],
> fut=GridCacheTxRecoveryFuture [trackable=true,
> futId=81c3b7af571-1093b7fe-20ae-4c3f-9adb-4ecac23c136e,
> tx=GridDhtTxLocal [nearNodeId=a7eded9b-4078-4ee5-a1dd-426b8debc203,
> nearFutId=e0576afd571-dbd82c53-1772-4c53-a4ea-38e601002379,
> nearMiniId=1, nearFinFutId=null, nearFinMiniId=0,
> nearXidVer=GridCacheVersion [topVer=216485010, order=1607062821327,
> nodeOrder=30], lb=null, super=GridDhtTxLocalAdapter
> [nearOnOriginatingNode=false, nearNodes=KeySetView [],
> dhtNodes=KeySetView [e4d4fc27-d2d9-47f9-8d21-dfac2c003b55,
> 3060fc02-e94a-4b6d-851a-05d75ea751e0], explicitLock=false,
> super=IgniteTxLocalAdapter [completedBase=null,
> sndTransformedVals=false, depEnabled=false,
> txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false,
> useMvccCaching=false], super=IgniteTxAdapter [xidVer=GridCacheVersion
> [topVer=216485010, order=1607062856849, nodeOrder=1],
> writeVer=GridCacheVersion [topVer=216485023, order=1607062856850,
> nodeOrder=1], implicit=true, loc=true, threadId=24070,
> startTime=1605506134277, nodeId=2b0db4f4-86d1-42c2-babf-f6318bd932e5,
> startVer=GridCacheVersion [topVer=216485010, order=1607062856849,
> nodeOrder=1], endVer=null, isolation=READ_COMMITTED,
> concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false,
> plc=2, commitVer=null, finalizing=RECOVERY_FINISH, invalidParts=null,
> state=PREPARED, timedOut=false, topVer=AffinityTopologyVersion
> [topVer=117, minorTopVer=0], mvccSnapshot=null, skipCompletedVers=false,
> parentTx=null, duration=370668ms, onePhaseCommit=false], size=1]]],
> failedNodeIds=SingletonSet [a7eded9b-4078-4ee5-a1dd-426b8debc203],
> nearTxCheck=false, innerFuts=EmptyList [],
> super=GridCompoundIdentityFuture [super=GridCompoundFuture [rdc=Bool
> reducer: true, initFlag=0, lsnrCalls=0, done=false, cancelled=false,
> err=null, futs=EmptyList []
> [2020-11-16 14:01:44,947][INFO ][sys-stripe-8-#9][IgniteTxManager]
> Finishing prepared transaction [commit=true, tx=GridDhtTxLocal
> [nearNodeId=a7eded9b-4078-4ee5-a1dd-426b8debc203,
> nearFutId=e0576afd571-dbd82c53-1772-4c53-a4ea-38e601002379,
> nearMiniId=1, nearFinFutId=null, nearFinMiniId=0,
> nearXidVer=GridCacheVersion [topVer=216485010, order=1607062821327,
> nodeOrder=30], lb=null, super=GridDhtTxLocalAdapter
> [nearOnOriginatingNode=false, nearNodes=KeySetView [],
> dhtNodes=KeySetView [e4d4fc27-d2d9-47f9-8d21-dfac2c003b55,
> 3060fc02-e94a-4b6d-851a-05d75ea751e0], explicitLock=false,
> super=IgniteTxLocalAdapter [completedBase=null,
> sndTransformedVals=false, depEnabled=false,
> txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false,
> useMvccCaching=false], super=IgniteTxAdapter [xidVer=GridCacheVersion
> [topVer=216485010, order=1607062856849, nodeOrder=1],
> writeVer=GridCacheVersion [topVer=216485023, order=1607062856850,
> nodeOrder=1], implicit=true, loc=true, threadId=24070,
> startTime=1605506134277, nodeId=2b0db4f4-86d1-42c2-babf-f6318bd932e5,
> startVer=GridCacheVersion [topVer=216485010, 

Re: Configuration Files and storage location

[Ilya Kasnacheev]

Hello!

Too many unrelated issues are wrapped together here.

First, you do not need to have identical config for all nodes in the
cluster. Paths may surely differ. Some property need to be the same, some
don't.

Second, you may change the node's consistentId to specify which persistence
files to use, even if their config is otherwise identical. If consistentId
is not used then one node may capture the data dir and the second one will
not be able to use it / create one from scratch.

Last, nodes may be added to cluster at any time but you should make sure
that baseline topology is fixed also. Node is not added/removed to baseline
topology unless you do that manually. You may also configure auto-adjust.

Regards,
-- 
Ilya Kasnacheev


пт, 20 нояб. 2020 г. в 13:08, Wolfgang Meyerle <
wolfgang.meye...@googlemail.com>:

> Hi community,
>
> I have a question regarding the xml configuration which is used in
> clients / servers and thinclients.
>
> Is it really necessary that each node is using the same configuration?
> I'm just asking because I'm trying out to setup multiple servers in one
> machine which totally is nonsense in a production release but worth
> working in a debugging environment and to find out how Ignite acts in
> certain use cases.
>
> I started basically two server configurations. One from a cpp code
> environment and the other from the usual ./ignite.sh bash script which
> has been provided with the release.
>
> The ignite environment was started first and somehow seems to block the
> cpp code from inserting data into the persistent storage as I cannot see
> my log output but just that the node has started up.
>
> After I started the ignite.sh script I activated the server node with
> control.sh --activate
>
> Is that maybe the problem as new nodes are not accepted anymore to be
> joined to the topology?
>
> Why do I have to activate the nodes or deactivate them anyway in the
> first place. In my mind the servers should be shutdown when the last
> node leaves the topology
>
> Regards,
>
> Wolfgang
>

Re: Ignite Visor cmd cannot connect...

[Ilya Kasnacheev]

Hello!

I think that C++ node would enforce some binary configuration, so you will
have to repeat the exact one in the XML file for Visor CMD:
https://ignite.apache.org/docs/latest/cpp-specific/cpp-platform-interoperability

I recommend using control.sh or GGCC for Apache Ignite in place of Visor
CMD where possible, since it's not actively maintained.

Regards,
-- 
Ilya Kasnacheev


пт, 20 нояб. 2020 г. в 14:08, Wolfgang Meyerle <
wolfgang.meye...@googlemail.com>:

> Hi,
>
> I tried using
>
> ignitevisorcmd.sh -cfg=
>
> however I'm getting the following error message:
> Local node's binary configuration is not equal to remote node's binary
> configuration
>
>
> It further states:
> localBinaryCfg=null
> rmtBinaryCfg={globIdMapper ...
>
> I'm using the exact same xml configuration file.
>
> However for some stupid reason the visorcmd cannot connect
> I'm using cpp code which is instantiating the server node.
> The node seems to be running as I can use a thin client to connect to it
> and put and get values...
>
> Why?
>
> Regards,
>
> Wolfgang
>

Ignite Visor cmd cannot connect...

[Wolfgang Meyerle]

Hi,

I tried using

ignitevisorcmd.sh -cfg=

however I'm getting the following error message:
Local node's binary configuration is not equal to remote node's binary 
configuration



It further states:
localBinaryCfg=null
rmtBinaryCfg={globIdMapper ...

I'm using the exact same xml configuration file.

However for some stupid reason the visorcmd cannot connect
I'm using cpp code which is instantiating the server node.
The node seems to be running as I can use a thin client to connect to it 
and put and get values...


Why?

Regards,

Wolfgang

Re: Updating a field using a SQL subquery

[Surkov.Aleksandr]

Apparently, this functionality is not supported.

https://apacheignite-sql.readme.io/docs/how-ignite-sql-works#known-limitations




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Updating a field using a SQL subquery

[Surkov.Aleksandr]

Hello.
I am trying to update a field using a SQL subquery, but I get an incorrect
result. Do I need to create a bug or am I doing something wrong?
UpdateWithSubQueryTest.java

  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Configuration Files and storage location

[Wolfgang Meyerle]

Hi community,

I have a question regarding the xml configuration which is used in 
clients / servers and thinclients.


Is it really necessary that each node is using the same configuration?
I'm just asking because I'm trying out to setup multiple servers in one 
machine which totally is nonsense in a production release but worth 
working in a debugging environment and to find out how Ignite acts in 
certain use cases.


I started basically two server configurations. One from a cpp code 
environment and the other from the usual ./ignite.sh bash script which 
has been provided with the release.


The ignite environment was started first and somehow seems to block the 
cpp code from inserting data into the persistent storage as I cannot see 
my log output but just that the node has started up.


After I started the ignite.sh script I activated the server node with 
control.sh --activate


Is that maybe the problem as new nodes are not accepted anymore to be 
joined to the topology?


Why do I have to activate the nodes or deactivate them anyway in the 
first place. In my mind the servers should be shutdown when the last 
node leaves the topology


Regards,

Wolfgang

Re: How to get data for subqueries with aggregate functions in Ignite Thick client mode

[Konstantin Orlov]

Hi Shravya!

It caused by bug in H2 parser which erases information about type of the column 
expression of subqueries and reports it just like ‘column’, so Ignite can’t 
properly split the query.
For thin client the splitting is not always needed since the requests starts 
its execution on the server node (i.g. in case when you execute query over a 
replicated cache). But for thick client the splitting is always needed.

The possible workaround is to add an alias to the column with aggregate inside 
of the subquery:
SELECT * FROM (select region, sum(total_cost) as sum FROM ecom_sales group by 
region limit 200) AS _X LIMIT 30 OFFSET 0
Hope it helps.

-- 
Regards,
Konstantin Orlov
Software Engineer, St. Petersburg
+7 (921) 445-65-75
https://www.gridgain.com
Powered by Apache® Ignite™



> On 3 Nov 2020, at 08:12, Shravya Nethula 
>  wrote:
> 
> Hi,
> 
> For Ignite thick client, the below sql query is throwing CacheException with 
> the following code:
> public class ClientNode {
> public static void main(String[] args) {
> IgniteConfiguration igniteCfg = new IgniteConfiguration();
> igniteCfg.setClientMode(true);
> 
> Ignite ignite = Ignition.start(igniteCfg);
> IgniteCache foo = ignite.getOrCreateCache("foo");
> SqlFieldsQuery sql = new SqlFieldsQuery("SELECT * FROM (select 
> region, sum(total_cost) FROM ecom_sales group by region limit 200) AS _X 
> LIMIT 30 OFFSET 0");
> List> data = foo.query(sql).getAll();
>  System.out.println("data:::"+data);
>  } } 
> 
> Output:
> Exception in thread "main" javax.cache.CacheException: Failed to parse query. 
> Column "_X__Z1.SUM(TOTAL_COST)" not found;
> 
> 
> On the other hand, for thin client, the result for the same sql query is 
> coming up as expected.
> public class ClientNode {
> public static void main(String[] args) {
> ClientConfiguration clientConfig = new ClientConfiguration();
> cc.setUserName("username");
> cc.setUserPassword("password");
> 
> IgniteClient thinClient = Ignition.startClient(clientConfig);
> SqlFieldsQuery sql = new SqlFieldsQuery("SELECT * FROM (select 
> region, sum(total_cost) FROM ecom_sales group by region limit 200) AS _X 
> LIMIT 30 OFFSET 0");
> List> data = thinClient.query(sql).getAll();
>System.out.println("data:::"+data);
>  } } 
> Output:
> [Europe, 2.4160571432339935E10]
> [Asia, 1.3585889703059944E10]
> 
> Such behaviour is observed only in the scenarios with subqueries having 
> aggregate functions. 
> Can you please explain, why is the behaviour different for the same query? 
> What can be changed to get the data properly even in Thick client mode?
> 
> Any help is much appreciated!
> Thanks in advance!
> 
> 
> Regards,
> Shravya Nethula,
> BigData Developer,
> 
> Hyderabad.