Re: Join between replicated and partitioned caches
Thanks. How about a join between two partitioned tables with different affinity keys or a join between two tables with one having an affinity key defined and the other with no affinity key explicitly defined? On Tue, Jan 19, 2021 at 11:59 AM Craig Gresbrink wrote: > Since a replicated cache has an entire copy of all its data on each server > node, the data is local so you can join to it is my understanding. > > > > *From:* Clay Teahouse > *Sent:* Tuesday, January 19, 2021 9:49 AM > *To:* user@ignite.apache.org > *Subject:* Join between replicated and partitioned caches > > > > *CAUTION:* This email originated outside 24 Hour Fitness. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. > > > > Hello All, > > > > I am using SqlFieldsQuery to execute sql queries and I am setting the > option setLocal to true to process the local data only. What happens if the > sql has a join with a replicated cache? > > > > thank you > > Clay >
Join between replicated and partitioned caches
Hello All, I am using SqlFieldsQuery to execute sql queries and I am setting the option setLocal to true to process the local data only. What happens if the sql has a join with a replicated cache? thank you Clay
Re: embedded jetty & ignite
Understood. Thank you, Denis for the explanation. On Mon, Jun 8, 2020 at 2:08 PM Denis Magda wrote: > Clay, > > I assume embedding jetty as an ignite service, i.e., as an ignite server >> node, is not desirable (still don't know the reason). > > > Let's suppose that your cluster has 3 server/data nodes with application > records distributed evenly - so that each node keeps ~33% of the whole data > set. Next, if to deploy Jetty as an ignite service on those 3 nodes and to > assume that all application records are accessed frequently, then for ~33% > of requests the service will serve local data of its node while for the > rest ~66% request it will getting it from the other 2 remote nodes. So, the > performance advantages of this deployment strategy are insignificant > (unless all the data is fully replicated). > > Also, if an Ignite service is deployed on a server node, then each time > you need to update service logic you'll restart the server node and trigger > cluster rebalancing that might be a time-consuming operation depending on > data size. This might be a more prominent implication that you would not > want to have in the production environment. > > > > - > Denis > > > On Sun, Jun 7, 2020 at 3:04 AM Clay Teahouse > wrote: > >> Denis -- Thank you for the recommendation. >> From what I read, embedded ignite servers has implications for >> scalability, and write performance and I cannot afford either. >> I assume embedding jetty as an ignite service, i.e., as an ignite server >> node, is not desirable (still don't know the reason). >> Yes, ultimatley my goal is to deploy the ignite services and data nodes >> as microservices. >> >> On Sat, Jun 6, 2020 at 11:25 PM Denis Magda wrote: >> >>> Clay, >>> >>> Probably, such frameworks as Quarkus, Spring Boot, Micronaut would work >>> as a better foundation for your microservices. As you know, those already >>> go with embedded REST servers and you can always use Ignite clients to >>> reach out to the cluster. >>> >>> Usually, Ignite servers are deployed in the embedded mode when you're >>> dealing with ultra-low latency use case or doing web-sessions clustering: >>> >>> https://www.gridgain.com/docs/latest/installation-guide/deployment-modes#embedded-deployment >>> >>> >>> - >>> Denis >>> >>> >>> On Sat, Jun 6, 2020 at 9:03 AM Clay Teahouse >>> wrote: >>> >>>> Hi Denis -- My main reason was for embedding jetty as an ignite service >>>> was to have ignite manage jetty instance, the same it does for any other >>>> kind of service. >>>> >>>> On Thu, Jun 4, 2020 at 3:30 PM Denis Magda wrote: >>>> >>>>> Clay, >>>>> >>>>> Do you have any specific requirements in mind for the ignite service + >>>>> jetty deployment? If possible, please tell us a bit more about your >>>>> application. >>>>> >>>>> Generally, I would deploy Jetty separately and use load balancers when >>>>> several instances of an application are needed. >>>>> >>>>> - >>>>> Denis >>>>> >>>>> >>>>> On Wed, Jun 3, 2020 at 3:20 PM Clay Teahouse >>>>> wrote: >>>>> >>>>>> Thank you, Denis. I'll research this topic further. >>>>>> >>>>>> Any recommendation for/against using jetty as an embedded servlet >>>>>> container, in this case, say, deployed as an ignite service? >>>>>> >>>>>> On Fri, May 29, 2020 at 11:22 PM Denis Magda >>>>>> wrote: >>>>>> >>>>>>> Clay, >>>>>>> >>>>>>> Just start your Jetty server and deploy as many instances of your >>>>>>> web app as needed. Inside the logic of those apps start Ignite server >>>>>>> nodes >>>>>>> instances. Then, refer to this documentation page for session clustering >>>>>>> configuration: >>>>>>> https://apacheignite-mix.readme.io/docs/web-session-clustering >>>>>>> >>>>>>> Also, there were many related questions related to this topic. Try >>>>>>> to search for specific by googling for "session clustering with ignite >>>>>>> and >>>>>>> jetty". >>>>>>> >>>>>&
Re: embedded jetty & ignite
Denis -- Thank you for the recommendation. >From what I read, embedded ignite servers has implications for scalability, and write performance and I cannot afford either. I assume embedding jetty as an ignite service, i.e., as an ignite server node, is not desirable (still don't know the reason). Yes, ultimatley my goal is to deploy the ignite services and data nodes as microservices. On Sat, Jun 6, 2020 at 11:25 PM Denis Magda wrote: > Clay, > > Probably, such frameworks as Quarkus, Spring Boot, Micronaut would work as > a better foundation for your microservices. As you know, those already go > with embedded REST servers and you can always use Ignite clients to > reach out to the cluster. > > Usually, Ignite servers are deployed in the embedded mode when you're > dealing with ultra-low latency use case or doing web-sessions clustering: > > https://www.gridgain.com/docs/latest/installation-guide/deployment-modes#embedded-deployment > > > - > Denis > > > On Sat, Jun 6, 2020 at 9:03 AM Clay Teahouse > wrote: > >> Hi Denis -- My main reason was for embedding jetty as an ignite service >> was to have ignite manage jetty instance, the same it does for any other >> kind of service. >> >> On Thu, Jun 4, 2020 at 3:30 PM Denis Magda wrote: >> >>> Clay, >>> >>> Do you have any specific requirements in mind for the ignite service + >>> jetty deployment? If possible, please tell us a bit more about your >>> application. >>> >>> Generally, I would deploy Jetty separately and use load balancers when >>> several instances of an application are needed. >>> >>> - >>> Denis >>> >>> >>> On Wed, Jun 3, 2020 at 3:20 PM Clay Teahouse >>> wrote: >>> >>>> Thank you, Denis. I'll research this topic further. >>>> >>>> Any recommendation for/against using jetty as an embedded servlet >>>> container, in this case, say, deployed as an ignite service? >>>> >>>> On Fri, May 29, 2020 at 11:22 PM Denis Magda wrote: >>>> >>>>> Clay, >>>>> >>>>> Just start your Jetty server and deploy as many instances of your web >>>>> app as needed. Inside the logic of those apps start Ignite server nodes >>>>> instances. Then, refer to this documentation page for session clustering >>>>> configuration: >>>>> https://apacheignite-mix.readme.io/docs/web-session-clustering >>>>> >>>>> Also, there were many related questions related to this topic. Try to >>>>> search for specific by googling for "session clustering with ignite and >>>>> jetty". >>>>> >>>>> Let us know if further help is needed. >>>>> >>>>> >>>>> - >>>>> Denis >>>>> >>>>> >>>>> On Fri, May 29, 2020 at 6:57 PM Clay Teahouse >>>>> wrote: >>>>> >>>>>> thank you Denis. >>>>>> If I want to go with the first option, how would I deploy jetty as >>>>>> embedded server? Do I deploy it as an ignite service? >>>>>> How would I do session clustering in this case? >>>>>> >>>>>> On Fri, May 29, 2020 at 3:18 PM Denis Magda >>>>>> wrote: >>>>>> >>>>>>> Hi Clay, >>>>>>> >>>>>>> I wouldn't suggest using Ignite's Jetty instance for the deployment >>>>>>> of your services. Ignite's Jetty primary function is to handle REST >>>>>>> requests specific to Ignite: >>>>>>> https://apacheignite.readme.io/docs/rest-api >>>>>>> >>>>>>> Instead, deploy and manage your restful services separately. Then, >>>>>>> if the goal is to do a web session clustering, deploy Ignite server >>>>>>> nodes >>>>>>> in the embedded mode making the sessions' caches replicated. Otherwise, >>>>>>> deploy the server nodes independently and reach the cluster out from the >>>>>>> restful services using existing Ignite APIs. This tutorial shows how to >>>>>>> do >>>>>>> the latter with Spring Boot: >>>>>>> https://www.gridgain.com/docs/tutorials/spring/spring_ignite_tutorial >>>>>>> >>>>>>> - >>>>>>> Denis >>>>>>> >>>>>>> >>>>>>> On Fri, May 29, 2020 at 8:25 AM Clay Teahouse < >>>>>>> clayteaho...@gmail.com> wrote: >>>>>>> >>>>>>>> hello, >>>>>>>> I understand that ignite comes with embedded jetty server. >>>>>>>> 1) Can I utilize this jetty server to deploy my own restful >>>>>>>> services (using Jersey implementation)? If yes, can you please direct >>>>>>>> me to >>>>>>>> some examples. >>>>>>>> Further questions: >>>>>>>> 2)How does the ignite embedded jetty work with regard to load >>>>>>>> balancing? Are there multiple instances of the embedded jetty server >>>>>>>> running behind a load balancer? In other words, can I invoke multiple >>>>>>>> instances? >>>>>>>> 2) How does this scheme work with web session clustering? >>>>>>>> 3) Would the ignite node run in server mode? >>>>>>>> 4) I want the jetty sessions access ignite caches (on the server >>>>>>>> side) as the data source for the data returned from the restful >>>>>>>> services. >>>>>>>> >>>>>>>> Any help and advice would be much appreciated. Thank you >>>>>>>> >>>>>>>
Re: embedded jetty & ignite
Hi Denis -- My main reason was for embedding jetty as an ignite service was to have ignite manage jetty instance, the same it does for any other kind of service. On Thu, Jun 4, 2020 at 3:30 PM Denis Magda wrote: > Clay, > > Do you have any specific requirements in mind for the ignite service + > jetty deployment? If possible, please tell us a bit more about your > application. > > Generally, I would deploy Jetty separately and use load balancers when > several instances of an application are needed. > > - > Denis > > > On Wed, Jun 3, 2020 at 3:20 PM Clay Teahouse > wrote: > >> Thank you, Denis. I'll research this topic further. >> >> Any recommendation for/against using jetty as an embedded servlet >> container, in this case, say, deployed as an ignite service? >> >> On Fri, May 29, 2020 at 11:22 PM Denis Magda wrote: >> >>> Clay, >>> >>> Just start your Jetty server and deploy as many instances of your web >>> app as needed. Inside the logic of those apps start Ignite server nodes >>> instances. Then, refer to this documentation page for session clustering >>> configuration: >>> https://apacheignite-mix.readme.io/docs/web-session-clustering >>> >>> Also, there were many related questions related to this topic. Try to >>> search for specific by googling for "session clustering with ignite and >>> jetty". >>> >>> Let us know if further help is needed. >>> >>> >>> - >>> Denis >>> >>> >>> On Fri, May 29, 2020 at 6:57 PM Clay Teahouse >>> wrote: >>> >>>> thank you Denis. >>>> If I want to go with the first option, how would I deploy jetty as >>>> embedded server? Do I deploy it as an ignite service? >>>> How would I do session clustering in this case? >>>> >>>> On Fri, May 29, 2020 at 3:18 PM Denis Magda wrote: >>>> >>>>> Hi Clay, >>>>> >>>>> I wouldn't suggest using Ignite's Jetty instance for the deployment of >>>>> your services. Ignite's Jetty primary function is to handle REST requests >>>>> specific to Ignite: https://apacheignite.readme.io/docs/rest-api >>>>> >>>>> Instead, deploy and manage your restful services separately. Then, if >>>>> the goal is to do a web session clustering, deploy Ignite server nodes in >>>>> the embedded mode making the sessions' caches replicated. Otherwise, >>>>> deploy >>>>> the server nodes independently and reach the cluster out from the restful >>>>> services using existing Ignite APIs. This tutorial shows how to do the >>>>> latter with Spring Boot: >>>>> https://www.gridgain.com/docs/tutorials/spring/spring_ignite_tutorial >>>>> >>>>> - >>>>> Denis >>>>> >>>>> >>>>> On Fri, May 29, 2020 at 8:25 AM Clay Teahouse >>>>> wrote: >>>>> >>>>>> hello, >>>>>> I understand that ignite comes with embedded jetty server. >>>>>> 1) Can I utilize this jetty server to deploy my own restful services >>>>>> (using Jersey implementation)? If yes, can you please direct me to some >>>>>> examples. >>>>>> Further questions: >>>>>> 2)How does the ignite embedded jetty work with regard to load >>>>>> balancing? Are there multiple instances of the embedded jetty server >>>>>> running behind a load balancer? In other words, can I invoke multiple >>>>>> instances? >>>>>> 2) How does this scheme work with web session clustering? >>>>>> 3) Would the ignite node run in server mode? >>>>>> 4) I want the jetty sessions access ignite caches (on the server >>>>>> side) as the data source for the data returned from the restful services. >>>>>> >>>>>> Any help and advice would be much appreciated. Thank you >>>>>> >>>>>
Re: embedded jetty & ignite
Thank you, Denis. I'll research this topic further. Any recommendation for/against using jetty as an embedded servlet container, in this case, say, deployed as an ignite service? On Fri, May 29, 2020 at 11:22 PM Denis Magda wrote: > Clay, > > Just start your Jetty server and deploy as many instances of your web app > as needed. Inside the logic of those apps start Ignite server nodes > instances. Then, refer to this documentation page for session clustering > configuration: > https://apacheignite-mix.readme.io/docs/web-session-clustering > > Also, there were many related questions related to this topic. Try to > search for specific by googling for "session clustering with ignite and > jetty". > > Let us know if further help is needed. > > > - > Denis > > > On Fri, May 29, 2020 at 6:57 PM Clay Teahouse > wrote: > >> thank you Denis. >> If I want to go with the first option, how would I deploy jetty as >> embedded server? Do I deploy it as an ignite service? >> How would I do session clustering in this case? >> >> On Fri, May 29, 2020 at 3:18 PM Denis Magda wrote: >> >>> Hi Clay, >>> >>> I wouldn't suggest using Ignite's Jetty instance for the deployment of >>> your services. Ignite's Jetty primary function is to handle REST requests >>> specific to Ignite: https://apacheignite.readme.io/docs/rest-api >>> >>> Instead, deploy and manage your restful services separately. Then, if >>> the goal is to do a web session clustering, deploy Ignite server nodes in >>> the embedded mode making the sessions' caches replicated. Otherwise, deploy >>> the server nodes independently and reach the cluster out from the restful >>> services using existing Ignite APIs. This tutorial shows how to do the >>> latter with Spring Boot: >>> https://www.gridgain.com/docs/tutorials/spring/spring_ignite_tutorial >>> >>> - >>> Denis >>> >>> >>> On Fri, May 29, 2020 at 8:25 AM Clay Teahouse >>> wrote: >>> >>>> hello, >>>> I understand that ignite comes with embedded jetty server. >>>> 1) Can I utilize this jetty server to deploy my own restful services >>>> (using Jersey implementation)? If yes, can you please direct me to some >>>> examples. >>>> Further questions: >>>> 2)How does the ignite embedded jetty work with regard to load >>>> balancing? Are there multiple instances of the embedded jetty server >>>> running behind a load balancer? In other words, can I invoke multiple >>>> instances? >>>> 2) How does this scheme work with web session clustering? >>>> 3) Would the ignite node run in server mode? >>>> 4) I want the jetty sessions access ignite caches (on the server side) >>>> as the data source for the data returned from the restful services. >>>> >>>> Any help and advice would be much appreciated. Thank you >>>> >>>
Re: connection refused
Thanks Wesley, Alex. Attached is a snapshot of the client log. I don't see anything on the ignite server side. I am/was trying to do get data from a replicated cache. On Tue, Jun 2, 2020 at 10:36 AM Alexandr Shapkin wrote: > Hi, > > > > As Wesley mentioned, It’s quite difficult to say something specific about > without the logs. > > > > As for the additional questions, can you explain, what Ignite API is being > used? Is it a Compute API call or > > a regular cache GETs? > > > > It’s totally fine to spawn several user threads and access the server > simultaneously. > > Regarding the node connectivity, by default there is a 1 connection per > node, you can control it > > using the TcpCommunicationSpi#setConnectionsPerNode [1] > > > > You can check the pool and adjust the sizes using the following docs [2]. > > If you perform much compute API calls, you might be interested in > increasing the server public pool size, > > striped pool for cache accesses and so on. > > > > In any case, it’s better to check your system resources utilization > (CPU/memory/etc) if it’s about 100% on a client > > and 20% on a server, you might want to add several more clients to a > cluster. > > > > [1] - > https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/communication/tcp/TcpCommunicationSpi.html#setConnectionsPerNode-int- > > > > [2] - https://apacheignite.readme.io/docs/thread-pools > > > > > > *From: *Clay Teahouse > *Sent: *Tuesday, June 2, 2020 5:20 PM > *To: *user@ignite.apache.org > *Subject: *Re: connection refused > > > > It turned out if I inject a pause of 4-5 seconds, then the client > connection goes through. > > Why the service grid requests go through immediately but the regular > cache access requires a pause from the time an ignite instance is acquired? > Subsequent calls are fast. The caches are replicated. > > Another question, probably trivial: > > Can you instantiate multiple concurrent connections from a client to the > compute nodes? Is the number of connections limited by the number of > threads? Or do I need to create separate client instances for each request? > > > > > > On Tue, Jun 2, 2020 at 5:46 AM Wesley Peng wrote: > > Does either client or server have any logs? > > > > Thanks > > > > On Tue, Jun 2, 2020 at 6:37 PM Clay Teahouse > wrote: > > I'd appreciate your help with this issue. > > I have a server and a client node, the latter running in tomcat. I get > connection refused if I try to connect to the server node from the client, > although the server shows the client has joined the cluster. I don't have a > problem running a service on the server's service grid from the client node. > > I don't have any issue connecting from any client anywhere, if the client > is not running in tomcat. > > what could be the problem? > > > > > log2 Description: Binary data
Re: connection refused
It turned out if I inject a pause of 4-5 seconds, then the client connection goes through. Why the service grid requests go through immediately but the regular cache access requires a pause from the time an ignite instance is acquired? Subsequent calls are fast. The caches are replicated. Another question, probably trivial: Can you instantiate multiple concurrent connections from a client to the compute nodes? Is the number of connections limited by the number of threads? Or do I need to create separate client instances for each request? On Tue, Jun 2, 2020 at 5:46 AM Wesley Peng wrote: > Does either client or server have any logs? > > Thanks > > On Tue, Jun 2, 2020 at 6:37 PM Clay Teahouse > wrote: > >> I'd appreciate your help with this issue. >> I have a server and a client node, the latter running in tomcat. I get >> connection refused if I try to connect to the server node from the client, >> although the server shows the client has joined the cluster. I don't have a >> problem running a service on the server's service grid from the client node. >> I don't have any issue connecting from any client anywhere, if the client >> is not running in tomcat. >> what could be the problem? >> >>
connection refused
I'd appreciate your help with this issue. I have a server and a client node, the latter running in tomcat. I get connection refused if I try to connect to the server node from the client, although the server shows the client has joined the cluster. I don't have a problem running a service on the server's service grid from the client node. I don't have any issue connecting from any client anywhere, if the client is not running in tomcat. what could be the problem?
Re: embedded jetty & ignite
thank you Denis. If I want to go with the first option, how would I deploy jetty as embedded server? Do I deploy it as an ignite service? How would I do session clustering in this case? On Fri, May 29, 2020 at 3:18 PM Denis Magda wrote: > Hi Clay, > > I wouldn't suggest using Ignite's Jetty instance for the deployment of > your services. Ignite's Jetty primary function is to handle REST requests > specific to Ignite: https://apacheignite.readme.io/docs/rest-api > > Instead, deploy and manage your restful services separately. Then, if the > goal is to do a web session clustering, deploy Ignite server nodes in the > embedded mode making the sessions' caches replicated. Otherwise, deploy the > server nodes independently and reach the cluster out from the restful > services using existing Ignite APIs. This tutorial shows how to do the > latter with Spring Boot: > https://www.gridgain.com/docs/tutorials/spring/spring_ignite_tutorial > > - > Denis > > > On Fri, May 29, 2020 at 8:25 AM Clay Teahouse > wrote: > >> hello, >> I understand that ignite comes with embedded jetty server. >> 1) Can I utilize this jetty server to deploy my own restful services >> (using Jersey implementation)? If yes, can you please direct me to some >> examples. >> Further questions: >> 2)How does the ignite embedded jetty work with regard to load balancing? >> Are there multiple instances of the embedded jetty server running behind a >> load balancer? In other words, can I invoke multiple instances? >> 2) How does this scheme work with web session clustering? >> 3) Would the ignite node run in server mode? >> 4) I want the jetty sessions access ignite caches (on the server side) as >> the data source for the data returned from the restful services. >> >> Any help and advice would be much appreciated. Thank you >> >
embedded jetty & ignite
hello, I understand that ignite comes with embedded jetty server. 1) Can I utilize this jetty server to deploy my own restful services (using Jersey implementation)? If yes, can you please direct me to some examples. Further questions: 2)How does the ignite embedded jetty work with regard to load balancing? Are there multiple instances of the embedded jetty server running behind a load balancer? In other words, can I invoke multiple instances? 2) How does this scheme work with web session clustering? 3) Would the ignite node run in server mode? 4) I want the jetty sessions access ignite caches (on the server side) as the data source for the data returned from the restful services. Any help and advice would be much appreciated. Thank you
Re: Ignite for Configuration Database
Thanks Denis for the feedback. Currently the plan is to use either mostly rest API and in some cases SQL. On Wed, Jan 8, 2020 at 6:39 PM Denis Magda wrote: > Clay, thanks for clarifying. That's Ignite's use case. I'm aware of some > companies that use Ignite similarly but cannot disclose the names. > > How are you planning to query data - SQL, compute tasks any other APIs? > > - > Denis > > > On Wed, Jan 8, 2020 at 4:29 PM Clay Teahouse > wrote: > >> Hello Denis, >> >> I mean a database that hosts the information about sites (e.g., hotels) >> and devices located the sites (e.g., hardware, software, configuration, >> labels, geolocation), there can be 1000s of sites with each site having >> many devices, totaling 100s of 1000s entries. The data is not volatile and >> we don't have to keep track of the changes in the same database. >> >> thanks >> Clay >> >> On Wed, Jan 8, 2020 at 3:12 PM Denis Magda wrote: >> >>> Hi Clay, >>> >>> Could you please elaborate on the technical requirements of this usage >>> scenario? >>> >>> - >>> Denis >>> >>> >>> On Wed, Jan 8, 2020 at 8:30 AM Clay Teahouse >>> wrote: >>> >>>> Hello All, >>>> >>>> What are the pros and cons of using ignite as an asset and >>>> configuration database? >>>> >>>> thanks, >>>> Clay >>>> >>>
Re: Ignite for Configuration Database
Hello Denis, I mean a database that hosts the information about sites (e.g., hotels) and devices located the sites (e.g., hardware, software, configuration, labels, geolocation), there can be 1000s of sites with each site having many devices, totaling 100s of 1000s entries. The data is not volatile and we don't have to keep track of the changes in the same database. thanks Clay On Wed, Jan 8, 2020 at 3:12 PM Denis Magda wrote: > Hi Clay, > > Could you please elaborate on the technical requirements of this usage > scenario? > > - > Denis > > > On Wed, Jan 8, 2020 at 8:30 AM Clay Teahouse > wrote: > >> Hello All, >> >> What are the pros and cons of using ignite as an asset and configuration >> database? >> >> thanks, >> Clay >> >
Ignite for Configuration Database
Hello All, What are the pros and cons of using ignite as an asset and configuration database? thanks, Clay
Re: max number of TCP connections
Sorry for the mix-up. On Wed, Jun 19, 2019 at 2:35 AM Stephen Darlington < stephen.darling...@gridgain.com> wrote: > This is the Apache Ignite user mailing list. For help with Apache Nifi > you’ll need to engage with their community. See here: > > https://nifi.apache.org/mailing_lists.html > > Regards, > Stephen > > On 18 Jun 2019, at 17:02, Clay Teahouse wrote: > > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListenSyslog/ > > On Tue, Jun 18, 2019 at 9:59 AM Ilya Kasnacheev > wrote: > >> Hello! >> >> What's ListenSyslog? I have not heard about it. >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> вт, 18 июн. 2019 г. в 06:24, Clay Teahouse : >> >>> Hi All >>> >>> What is the max number of TCP connections that can be specified for the >>> ListenSyslog processor? The default seems to be 2. >>> >>> thanks >>> Clay >>> >>> > >
Re: max number of TCP connections
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListenSyslog/ On Tue, Jun 18, 2019 at 9:59 AM Ilya Kasnacheev wrote: > Hello! > > What's ListenSyslog? I have not heard about it. > > Regards, > -- > Ilya Kasnacheev > > > вт, 18 июн. 2019 г. в 06:24, Clay Teahouse : > >> Hi All >> >> What is the max number of TCP connections that can be specified for the >> ListenSyslog processor? The default seems to be 2. >> >> thanks >> Clay >> >>
max number of TCP connections
Hi All What is the max number of TCP connections that can be specified for the ListenSyslog processor? The default seems to be 2. thanks Clay
sizing
Hi All If I choose binary data type for my objects, as opposed to varchar, will it result in any saving, and yes, how much? I know that binary type would be faster to read/write but wanted to see if there will be any saving in storage. thanks Clay
views in ignite
Hello All, Is it possible to define database/table views in ingite? thanks Clay
Re: Populating tables via IgniteDataStreamer
Thank you for the replies. Any example, with using a cache template for creating the tables and using the streamer API to load the table? Also, one more thing: is there a performance penalty for using the relational structure imposed on the data, as opposed to manipulating the java objects directly? If no, what is the most optimized way of populating and accessing the tables? thank you again Clay On Tue, Feb 19, 2019 at 3:11 AM Stephen Darlington < stephen.darling...@gridgain.com> wrote: > Ignite comes with a nice sample for using the Data Streamer API: > https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/streaming/StreamTransformerExample.java > > > There’s nothing really “special” about caches/tables created using DDL > rather than one of the other methods. > > Regards, > Stephen > > On 19 Feb 2019, at 03:49, Clay Teahouse wrote: > > Hello All, > > Can someone share an example of utilizing IgniteDataStreamer API to > populate a table created via a DDL? > > thank you, > Clay > > > >
Populating tables via IgniteDataStreamer
Hello All, Can someone share an example of utilizing IgniteDataStreamer API to populate a table created via a DDL? thank you, Clay
Re: ignite questions
Thanks for everyone's feedback regarding the capacity planning question. My main objective here is to size my servers accordingly and keep the related data on the same server, as much as possible. It seems a custom affinity function that takes into account the server classes (defined based on capacity) is the potential solution as suggested. Naveen -- I will keep in mind your suggestion as well, regarding partitioning the huge data sets. thanks. Any feedback regarding my other questions: Data pin to cache: How do I make sure certain data never gets evicted (with native persistence enabled)? For example, I want my dimension data to always stay in cache. How do I implement service pipelining in apache ignite? Would continuous query be the mechanism? Any examples? Streaming: Are there examples on how to define watermarks, i.e., input completeness with regard to the event timestamp? On Fri, Jan 4, 2019 at 3:33 AM Naveen wrote: > Regarding your question on capacity planning > > Not sure we have any work around to have data equally getting distributed > to > fulfill your requirement technically and do the sizing. But > non-technically, > you can change your design to include state as well as part of your > affinity > key along with the country and rest of the fields you have, so that data > pertaining to India can be segregated into chunks of data which is state > wise and if you see statewide data is definitely not so huge compared to > country as a whole and same time relevant data is also getting stored on a > single node. Your design also should permit this to keep state name as > part > of your affinity key to resolve this use case. > > > Thanks > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: ignite questions
Thanks Mikael. I did come across that link before, but I am not sure it addresses my concern. I want to see how I need I size my physical VMs based on affinity keys. How would I say for India affinity key use this super size VM and for others use the other smaller ones, so the data doesn't get shuffled around? Maybe, there is no way, and I just have to wait for ignite to rebalance the partitions and fit things where they should be based on the affinity key. On Wed, Jan 2, 2019 at 8:32 AM Mikael wrote: > You can find some information about capacity planning here: > > https://apacheignite.readme.io/docs/capacity-planning > > About your India example you can use affinity keys to keep data together > in groups to avoid network traffic. > > https://apacheignite.readme.io/docs/affinity-collocation > > Mikael > Den 2019-01-02 kl. 14:44, skrev Clay Teahouse: > > Thanks Naveen. > > -- Cache Groups: When would I start considering cache groups, if my system > is growing, and sooner or later I will have to add to my caches and I need > to know 1) should I starting grouping now (I'd think yes), 2) if no, when, > what number of caches? > -- Capacity Planning: So, there is no guidelines on how to size the nodes > and the physical storage nodes reside on? How do I make sure all the > related data fit the same VM? It can't be the case that I have to come up > with 100s of super size VMs just because I have one instance with a huge > set of entries. For example, if I have millions of entries for India and > only a few for other countries, how do I make sure all the India related > data fits the same VM (to avoid the network) and have the data for all the > small countries fit on the same VM? > -- Pinning the data to cache: the data pinned to on-heap cache does not > get evicted from the memory? I want to see if there is something similar to > Oracle's memory pinning. > -- Read through: How do I know if something on cache or disk (using native > persistence)? > 5) Service chaining: Is there an example of service chaining that you can > point me to? > > 6) How do I implement service pipelining in apache ignite? Would > continuous query be the mechanism? Any examples? > > 7) Streaming: Are there examples on how to define watermarks, i.e., input > completeness with regard to the event timestamp? > > thank you > Clay > > On Tue, Jan 1, 2019 at 11:29 PM Naveen wrote: > >> Hello >> Couple of things I would like to with my experience >> >> 1. Cache Groups : Around 100 caches, I do not think we need to go for >> Cache >> groups, as you mentioned cache groups will have impact on you read/writes. >> However, changing the partition count to 128 from default 1024 would >> improve >> your cluster restart. >> >> 2. I doubt if Ignite has any settings we have for this. >> >> 3. The only I can think of is to keep the data in on-heap if the data size >> is not so huge. >> >> 4. Read through, with native persistence enabled, doing a read to the disk >> will load the cache. But the read is much slower compared with read from >> RAM, by default it does not pre-load the data. If you want to avoid this >> you >> can pre-load the data programatically and load Memory, good for even SQL >> SELECT as well. But with the 3rd party persistence, we need to pre-load >> the >> data to make your read work for SQL SELECT. >> >> Thanks >> Naveen >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >
Re: ignite questions
Thanks Naveen. -- Cache Groups: When would I start considering cache groups, if my system is growing, and sooner or later I will have to add to my caches and I need to know 1) should I starting grouping now (I'd think yes), 2) if no, when, what number of caches? -- Capacity Planning: So, there is no guidelines on how to size the nodes and the physical storage nodes reside on? How do I make sure all the related data fit the same VM? It can't be the case that I have to come up with 100s of super size VMs just because I have one instance with a huge set of entries. For example, if I have millions of entries for India and only a few for other countries, how do I make sure all the India related data fits the same VM (to avoid the network) and have the data for all the small countries fit on the same VM? -- Pinning the data to cache: the data pinned to on-heap cache does not get evicted from the memory? I want to see if there is something similar to Oracle's memory pinning. -- Read through: How do I know if something on cache or disk (using native persistence)? 5) Service chaining: Is there an example of service chaining that you can point me to? 6) How do I implement service pipelining in apache ignite? Would continuous query be the mechanism? Any examples? 7) Streaming: Are there examples on how to define watermarks, i.e., input completeness with regard to the event timestamp? thank you Clay On Tue, Jan 1, 2019 at 11:29 PM Naveen wrote: > Hello > Couple of things I would like to with my experience > > 1. Cache Groups : Around 100 caches, I do not think we need to go for Cache > groups, as you mentioned cache groups will have impact on you read/writes. > However, changing the partition count to 128 from default 1024 would > improve > your cluster restart. > > 2. I doubt if Ignite has any settings we have for this. > > 3. The only I can think of is to keep the data in on-heap if the data size > is not so huge. > > 4. Read through, with native persistence enabled, doing a read to the disk > will load the cache. But the read is much slower compared with read from > RAM, by default it does not pre-load the data. If you want to avoid this > you > can pre-load the data programatically and load Memory, good for even SQL > SELECT as well. But with the 3rd party persistence, we need to pre-load the > data to make your read work for SQL SELECT. > > Thanks > Naveen > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: ignite questions
I'd have at least 80-90 caches, assuming each cache corresponds to a table. Any feedback regarding other questions? thanks, Clay On Mon, Dec 31, 2018 at 9:58 AM Mikael wrote: > Hi! > > You have to try, if you just have a few caches (<10) you may not need to > go for any cache groups at all but the more caches you have the more the > need for cache group pops up, it will create lots of file handles and > use lots of memory that can be kept under control with cache groups. > > Mikael > > Den 2018-12-31 kl. 14:01, skrev Clay Teahouse: > > Hello All, > > I am new to ignite and have several general questions. I'd appreciate > > your feedback. > > > > 1) Cache groups: according to the ignite documentation, cache groups > > help with scaling and performance but might hurt reads. Where is the > > balance? > > > > 2) Capacity planning: If I reading the docs correctly, with native > > persistence enabled, you do not need to specify cache eviction. If so, > > assuming I have data and compute affinity enabled, how do I size my > > nodes, to make sure my data stays in cache, considering volume > > discrepancy in different class of data? Say for example, I have the > > data for Canada and India, with India data being 10 times the data for > > Canada. How do I size my nodes to make sure the last month data for > > India and Canada stay in cache? > > > > 3) Data pin to cache: How do I make sure certain data never gets > > evicted (with native persistence enabled)? For example, I want my > > dimension data to always stay in cache. > > > > 4) Read through: If I am using native persistence, do I need to > > explicitly load the cache, once the data is on disk and no longer in > > cache, or doing a read to the data on the disk, will load the cache? > > If yes, is that true about SQL select as well? Is this possible with > > 3rd party persistence as well, say, postgresql. > > > > 5) Service chaining: Is there an example of service chaining that you > > can point me to? > > > > 6) How do I implement service pipelining in apache ignite? Would > > continuous query be the mechanism? Any examples? > > > > 7) Streaming: Are there examples on how to define watermarks, i.e., > > input completeness with regard to the event timestamp? > > > > > > > > thank you, > > Clay >
ignite questions
Hello All, I am new to ignite and have several general questions. I'd appreciate your feedback. 1) Cache groups: according to the ignite documentation, cache groups help with scaling and performance but might hurt reads. Where is the balance? 2) Capacity planning: If I reading the docs correctly, with native persistence enabled, you do not need to specify cache eviction. If so, assuming I have data and compute affinity enabled, how do I size my nodes, to make sure my data stays in cache, considering volume discrepancy in different class of data? Say for example, I have the data for Canada and India, with India data being 10 times the data for Canada. How do I size my nodes to make sure the last month data for India and Canada stay in cache? 3) Data pin to cache: How do I make sure certain data never gets evicted (with native persistence enabled)? For example, I want my dimension data to always stay in cache. 4) Read through: If I am using native persistence, do I need to explicitly load the cache, once the data is on disk and no longer in cache, or doing a read to the data on the disk, will load the cache? If yes, is that true about SQL select as well? Is this possible with 3rd party persistence as well, say, postgresql. 5) Service chaining: Is there an example of service chaining that you can point me to? 6) How do I implement service pipelining in apache ignite? Would continuous query be the mechanism? Any examples? 7) Streaming: Are there examples on how to define watermarks, i.e., input completeness with regard to the event timestamp? thank you, Clay
