Re: rebalancing & K8

[Ilya Kasnacheev]

Hello!

Usually you need to adjust (or auto-adjust) baseline topology after scaling
the cluster up or down.

You also need to make sure that nodes stay in one cluster.

Regards,
-- 
Ilya Kasnacheev


пн, 26 апр. 2021 г. в 15:02, narges saleh :

> Hi folks
>
> If I am deploying my ignite cluster using AKS, is defining the auto
> discovery service sufficient?
> I  am following this link:
>
> https://ignite.apache.org/docs/latest/installation/kubernetes/azure-deployment
>
> Specifically, I am concerned about ignite's node/partition rebalancing in
> case of auto-scaling. When K8 adds or removes nodes and pods, meaning,
> ignite nodes get added or removed, does rebalancing kicks in properly? Do I
> need to tune any parameter specifically for the purpose of deployment into
> K8? Do I need to set up liveness probes?
>
> thanks.
>

Re: Too many TCP discovery accepted incoming connections

[Ilya Kasnacheev]

Hello!

Please consider the following messages:

[2021-04-21T14:55:09,203][WARN
][tcp-comm-worker-#1%EDIFCustomer%][TcpCommunicationSpi] Connect timed out
(consider increasing 'failureDetectionTimeout' configuration property)
[addr=/10.40.0.78:47100, failureDetectionTimeout=6]
[2021-04-21T14:55:09,203][WARN
][tcp-comm-worker-#1%EDIFCustomer%][TcpCommunicationSpi] Failed to connect
to a remote node (make sure that destination node is alive and operating
system firewall is disabled on local and remote hosts) [addrs=[/
10.40.0.78:47100, /127.0.0.1:47100]]

I can see that communication threads will spend a lot of time on connect(),
indicating network or firewall issues:
Thread [name="tcp-comm-worker-#1%EDIFCustomer%", id=365, state=RUNNABLE,
blockCnt=1294, waitCnt=12569]
at sun.nio.ch.Net.poll(Native Method)
at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:954)
- locked java.lang.Object@65ec5b09
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:110)
- locked java.lang.Object@9ecd49c
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3299)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.access$6000(TcpCommunicationSpi.java:271)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:4489)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4294)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

I think this is the root cause. Your server node cannot connect to some of
your remaining nodes' communication port. Maybe your server node is behind
NAT or firewall. Consider enabling NAT traversal feature:
https://ignite.apache.org/docs/latest/clustering/running-client-nodes-behind-nat

Regards,
-- 
Ilya Kasnacheev


чт, 22 апр. 2021 г. в 21:58, Gangaiah Gundeboina :

> HI Ilya,
>
> Please find attached full log file.
>
> Regards,
> Gangaiah
>
> server_log.zip
> 
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Understanding SQL join performance

[Ilya Kasnacheev]

Hello!

If you had any images in your email, we are not seeing them. Please provide
links.

Regards,
-- 
Ilya Kasnacheev


сб, 24 апр. 2021 г. в 03:24, William.L :

> Hi,
>
> I am trying to understand why my colocated join between two tables/caches
> are taking so long compare to the individual table filters.
>
> TABLE1
>
> Returns 1 count -- 0.13s
>
> TABLE2
>
> Returns 65000 count -- 0.643s
>
>
>  JOIN TABLE1 and TABLE2
>
> Returns 650K count -- 7s
>
> Both analysis_input and analysis_output has index on (cohort_id, user_id,
> timestamp). The affinity key is user_id. How do I analyze the performance
> further?
>
> Here's the explain which does not tell me much:
>
>
>
> Is Ignite doing the join and filtering at each data node and then sending
> the 650K total rows to the reduce before aggregation? If so, is it possible
> for Ignite to do the some aggregation at the data node first and then send
> the first level aggregation results to the reducer?
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

[Community] Submit a talk to Ignite Summit until this Friday + Sneak Peek

[Kseniya Romanova]

Hi Igniters! If you planned to submit a talk to Ignite Summit someday, the
time has come! Call for Presentations is open until the end of the day
April 30, 11 PM PDT: https://sessionize.com/ignite-summit


Check the Summit Sneak Peek - you will be in good company!
https://ignite-summit.org/schedule


   - Shahir Daya, IBM Distinguished Engineer, and CTO - How IBM Leverages
   Event-Driven Architecture with Apache Ignite for Core Application
   Modernization
   - Emmanuel Wiesenfeld, BNP Paribas CIB Lead Developer  - Apache Ignite
   as a Hybrid Transactional-Analytical Processing Solution at a Large
   Investment Bank
   - Alexey Goncharuk, GridGain Chief Architect & Querify Labs Chief
   Researcher, Apache Ignite PMC - Apache Ignite Storage Engine Architecture:
   Tradeoffs and a Retrospective


Please contact me if you have any questions about the Summit registration
or submission.

-- 
Cheers,
Kseniya

Devrel at GridGain
Don't miss upcoming community events, join
https://www.meetup.com/Apache-Ignite-Virtual-Meetup/

Re: Designing Affinity Key for more locality

[Ilya Kasnacheev]

Hello!

SQL query planner will not understand the locality if you use surrogate
value as affinity key.

Maybe you need to define your own affinity function (extends
RendezvousAffinityFunction) which will map keys to partitions. I'm not sure
that it will help query planner though.

Regards,
-- 
Ilya Kasnacheev


вт, 27 апр. 2021 г. в 09:01, Pavel Tupitsyn :

> Hi William,
>
> Can you describe the use case and domain model in more detail?
>
> 1. AffinityKey is used to colocate some data with other data.
>What do you achieve with user-id being the affinity key?
>
> 2. If you'd like to put all users for a given tenant/group
> to the same node for efficiency, then use tenant-id as the user
> affinity key.
> UUID is fine, no need for extra logic with ints.
>
> On Tue, Apr 27, 2021 at 5:33 AM William.L  wrote:
>
>> Came across this statement in the Data Partitioning documents:
>>
>> "The affinity function determines the mapping between keys and partitions.
>> Each partition is identified by a number from a limited set (0 to 1023 by
>> default)."
>>
>> Looks like there is no point for adding another layer of mapping unless I
>> am
>> going for a smaller number.
>> Are there other ways in ignite to get more locality for subset of the
>> data?
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: [Meetup] Native persistence storage overview. April 27 2021

[Kseniya Romanova]

Thank you, Anton!
Here's the video https://youtu.be/LiUoZtSdJ3Y and slides https://t.ly/902X

пн, 19 апр. 2021 г. в 17:57, Anton,Kalashnikov :

> Hi Igniters,
>
> There is meetup next week, where I want to share my knowledge about
> Ignite Persistence module.  It will be high level overview of the main
> components and the main ideas behind the Persistence module. So if you
> are a new ignite contributor or you don't feel  confident enough in
> Ignite persistence it can be helpful for you.
>
> If you interested you can join at 8AM(PST), April 27. More info -
> https://www.meetup.com/ru-RU/Apache-Ignite-Virtual-Meetup/events/277298901
> .
>
> --
> Best regards,
> Anton Kalashnikov
>