Re: Inserting date into ignite with spark jdbc

, Denis Magda пишет: Andrey, Do we need to update our docs? It feels like the docs miss these details or have an outdated example. - Denis On Fri, Oct 30, 2020 at 7:03 AM Andrei Aleksandrov aealexsandrov@ mailto: aealexsandrov@ > wrote: Hi, Here's an example with correct syn

Re: Too long JVM pause out of nowhere leading into shutdowns of ignite-servers

Hi, Long JVM pauses can lead to different problems, but in your case, I see some network problems that lead to a segmentation of some nodes: Connection to Zookeeper server is lost, local node SEGMENTED. What you can do to avoid current problem: 1)You should find out the reason for the pause.

Re: Execution of local SqlFieldsQuery on client node disallowed

Hello, You can use events: https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/events/CacheRebalancingEvent.html But in this case, you should wait until the rebalancing of all caches is completed. Also you can use the web console tool:

Re: Inserting date into ignite with spark jdbc

Magda пишет: Andrey, Do we need to update our docs? It feels like the docs miss these details or have an outdated example. - Denis On Fri, Oct 30, 2020 at 7:03 AM Andrei Aleksandrov mailto:aealexsand...@gmail.com>> wrote: Hi, Here's an example with correct syntax that shoul

Re: Ignite Cluster Issue on 2.7.6

 and then tried to start cluster , but only 1 node is  joins cluster other node  does not participates in cluster. Do I have to add all nodes into BLT ? Regards, Gurmehar Singh *From:*Andrei Aleksandrov *Sent:* 29 October 2020 20:11 *To:* user@ignite.apache.org *Subject:* Re: Ignite Cluster

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

Hi, Often, problems with establishing a communication connection can be solved with the following configuration: 1)You may have multiple network interfaces and the wrong one could be used. Solved by changing the SPI communication timeouts.:    

Re: Ignite instances frequently failing - BUG: soft lockup - CPU#1 stuck

Hello, Too little information has been provided on your part: 1) Could you provide the screenshot from the web console at this time? 2) Could you collect Ignite logs during this period? 3) What tool shows that the processors are frozen? Have you checked other tools? BR, Andrew 10/30/2020

Re: Inserting date into ignite with spark jdbc

Hi, Here's an example with correct syntax that should work fine: |DataFrameWriter < Row > df = resultDF .write() .format(IgniteDataFrameSettings.FORMAT_IGNITE()) .option(IgniteDataFrameSettings.OPTION_CONFIG_FILE(), configPath) .option(IgniteDataFrameSettings.OPTION_TABLE(), "Person")

Re: Ignite Cluster Issue on 2.7.6

Hi, Do you use cluster with persistence? After first actication all your data will be located on the first activated node. In this case, you also should track your baseline. https://www.gridgain.com/docs/latest/developers-guide/baseline-topology Baseline topology is a subset of nodes where

Re: Large Heap with lots of BinaryMetaDataHolders

Hello, Let's start from the very beginning. 1) Could you please share the server and client config? 2) Java code of what you have in your client launcher application I will try to investigate your case. BR, Andrew 10/28/2020 7:19 PM, ssansoy пишет: Hi, could anyone please help understand

Re: High number of Exception Unexpected response ID with setTimeout

Hi, It will be great to take a look at full logs that contain these errors. BR, Andrei 9/28/2020 12:35 PM, AravindJP пишет: Thank you stepan for your reply , problem is the parent thread calling ignite cache get has a timeout of 120 ms , hence we allow a max of 100 ms for every cache read .

Re: OutOfMemoryException with Persistence and Eviction Enabled

Hi, It looks like that I found the issue: https://issues.apache.org/jira/browse/IGNITE-8917 When you use *put *or *removeAll *in persistence cache with more data than data region size throw IgniteOutOfMemoryException. Data streamer looks like don't affected by this ticket. The WA is pretty

Re: IgniteCache.size() is hanging

Hi, Highly likely some of the nodes go offline and try to connect again. Probably you had some network issues. I think I will see this and other information in the logs. Can you provide them? BR, Andrei 9/24/2020 6:54 PM, Alan Ward пишет: The only log I see is from one of the server nodes,

Re: OutOfMemoryException with Persistence and Eviction Enabled

Hi, Did you try to increase the DataRegion size a little bit? It looks like 190 MB isn't enough for some internal structures that Ignite stores in OFF-HEAP except the data. I suggest you increase the data region size to for example 512 MB - 1024 MB and take a look at how it will work. If

Re: Performance on Windows vs Linux

Got it. If everything is the same then it looks like I can't suggest something to you here but as I know most users run their solutions on Linux OS. Probably it can be related to some performance benefits of Linux OS but I don't think that this difference should be significant. 7/23/2020 7:34

Re: Performance on Windows vs Linux

Hi, I guess that you should take a look from another point of view. You have two servers that should be compared to each other. Take a look at the following things: 1)Disk speed and size 2)Network latency 3)CPU and RAM capacities I guess that these things will be more important than the

Re: 2.8.1 : Ignite Security : Cache_Put event generated from a remote_client user action has subject uuid of Node that executes the request

Hi Veena, Indeed it looks like that current problem wasn't solved. It looks like there are not enough people interested in the current fix. However, Ignite is the open-source community. You can make a patch for you or even provide it to the community. Unfortunately, I don't think that

Re: third-party persistance and junction table

Hi, Unfortunately, Ignite doesn't support such kind of relations out of the box. Ignite just translates it to third party data storage that used as cache-store. It's expected that inserts and updates will be rejected in case if they break some rules. BR, Andrei 7/21/2020 11:16 AM, Bastien

Re: Cache query exception when using generic type: class java.util.ArrayList cannot be cast to

Hi, You can put different types of objects in your cache because of the specific of its implementation. But this possibility can break your ScanQuery because you are going to see in cache only StationDto objects. I guess that previously you put ArrayList inside the cache and then you put

Re: cache.getAsync() blocks if cluster is not activated.

Hi, I don't think that it should hang because there are no cache operations allowed when the cluster isn't activated. It should be some kind of CacheExcdption. Is it possible to prepare some reproducer or unit test? Otherwise please provide some details: 1)What Ignite version was used?

Re: Ignite node log file setup

Hi, Can you please attach the whole log file? BR, Andrei 6/9/2020 5:14 AM, kay пишет: Hello! I start up sh ./ignite.sh -J-DgridName=testGridName -v ./config/config-cache.xml and in config-cache.xml but server start failed. Is that not proper to set igniteInstanceName?? log is here

Re: Cache_Put event generated from a remote_client user action has subject uuid of Node that executes the request sometimes

Hi, Your approach is correct as for me. IgniteCache can help to hold all the information about started transactions and users. Using EventStorageSpi is a good way to handle the events. BR, Andrei 6/9/2020 10:06 AM, VeenaMithare пишет: Cache_Put event generated from a remote_client user

Re: Ignite.cache.loadcache.Does this method do Increamental Load?

Hi, loadCache - will load all the keys and values from third party store into Ignite. It can be used for initial loading and for restore of the Ignite cache. There are no incremental updates. In case if you are going to do updates in the third party store not using Ignite API then you also

Re: How to access IGFS file written one node from other node in cluster ??

Hi, I can suggest to use the cache store implementation. For example current guide shows how Hive schema can be imported using web console. In case if you require to work with files (not

Re: Slow cache updates with indexing module enabled

Hi, Some recommendations should be applied to every SQL query: 1)You should avoid using __SCAN_ 2)You should use LAZY if the result set is big. 3)You should inline the indexes Please read my comments: 1)My question is what is happening with all those indexes when an entry is updated but, none

Re: Scheduling Cache Refresh using Ignite

Hi Nithin, You face current message because your client lost the connection to the server. It tries to get the acknowledge message on some operation (I guess it should be some cache operation). You can see that IgniteClientDisconnectedException was thrown. In this case, you can get the

Re: Ignite yarn resources keep on increasing

Hi, Could you please provide more details: 1)Your configurations and environment variables (IGNITE_PATH?) 2)The logs of your Ignite nodes where you see the mentioned exception. IGNITE_PATH - a path to unzipped Ignite distribution instead of the URL. Is it possible that you didn't unzip the

Re: Scheduling Cache Refresh using Ignite

Hi, Can you please attach the full logs with the mentioned exception? BTW I don't see any attaches in the previous message (probably user list can't do it). BR, Andrei 2/13/2020 3:44 PM, nithin91 пишет: Attached the bean file used -- Sent from:

Re: Slow cache updates with indexing module enabled

Hi, SQL query performance can be not great because of several cases: 1)Incorrect indexes. Please check that your EXPLAIN contains indexes and doesn't have scans for joins: INNER JOIN PUBLIC.PERSON P__Z1     /* PUBLIC.PERSON.__SCAN_ */ Probably the inline size for used index is incorrect or

Re: Scheduling Cache Refresh using Ignite

Hi, Please read my comments: 1)Ignite generally doesn't support changing of the cache configuration without re-creation of the the cache. But for SQL caches that were created via QueryEntity or CREATE TABLE you can add and remove the columns using ALTER TABLE commands:

Re: Ignite on yarn doesn't started

Hi, I asked you to check it because I see the next option: IGNITE_PATH = /tmp/ignite/apache-ignite-2.7.6-bin.zip This option should be a path to *unzipped *Ignite distribution instead of the URL (you set zip) Also I see commented IGNITE_URL option: #IGNITE_URL =

Re: Using EntryProcessor arguments recommendations

Hi, I suggest to read the documentation: EntryProcessor: https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheEntry.html Invoke java doc:

Re: JDBC thin client incorrect security context

Hi, I see that you found the ticket related to the current issue: https://issues.apache.org/jira/browse/IGNITE-12589 Looks like it can be a reason of your problem. Generally, I don't know how you implemented your security plugin if you take a look at similar plugin from third party vendor

Re: Issue with large number of Continuous Queries

Hi, Please read some comments below: 1)You said: *Scenario*: There are 1000 items in the myCache and client-1 is pushing 3 updates per second on every item. Lets say both client-2 and client-3 have 1000 different continuous queries open to listen to every update. Continues query can handle

Re: Ignite on yarn doesn't started

Duplicate of http://apache-ignite-users.70518.x6.nabble.com/Ignite-on-yarn-doesn-t-started-td31325.html 2/10/2020 11:00 PM, v.shinkevich пишет: Hi, All! I tried to run Ignite on Yarn My cluster properties: I also tried to add hdfs:// before paths, but result is the same. I created dir

Re: Ignite on yarn doesn't started

Hi, Can you attach the Ignite node logs? Also, I suggest to check: 1) check that Ignite libs (from ignite_binaries/libs) are available for your YARN deployment. 2) check that path to the configuration file is reachable from every node I see that you set absolute paths for to tmp folder.

Re: Exception when joining a data node

Hi, Are you sure that you saw this message on node stopping? It's strange because the following message should appear on node joining (and you can see it from the stack trace): BaselineTopology of joining node (bf40986d-fba5-4985-b128-d30bb45228e7) is not compatible with BaselineTopology in

Re: Cache.replace-K-V-V- performing slow

Hi, I have also described the possible problem here - https://stackoverflow.com/questions/59950157/why-apache-ignite-cache-replace-k-v-v-api-call-performing-slow BR, Andrei 1/31/2020 12:33 PM, Ilya Kasnacheev пишет: Hello! I'm not sure the benchmarking is relevant, since most of work

Re: "Adding entry to partition that is concurrently evicted" error

Hi, Current problem should be solved in ignite-2.8. I am not sure why this fix isn't a part of ignite-2.7.6. https://issues.apache.org/jira/browse/IGNITE-11127 Your cluster was stopped because of failure handler work.

Re: Distinct Query too slow

Hi, Looks like it's known issue: https://issues.apache.org/jira/browse/IGNITE-10781 According to this issue, indexes can work non-effective for distinct clause. However, looks like the explain from your log isn't full. It should contain two parts - reducer and mapper. Can you please run

Re: How to set timeout while use data streamer in sql mode.

Hi, You can find all existed options here: https://apacheignite-sql.readme.io/docs/jdbc-client-driver https://apacheignite-sql.readme.io/docs/jdbc-driver https://apacheignite-sql.readme.io/docs/odbc-driver At the moment I see that only *jdbc-client-driver* has special *streamingFlushFrequency

Re: CEP and Sliding Window Documentation

Hi, That information wasn't removed. It was moved to correct place: 1)Next section https://apacheignite.readme.io/v1.0/docs/data-streamers was moved herehttps://apacheignite.readme.io/docs/data-streamers

Re: Streamer and data loss

Hi, Not flushed data in a data streamer will be lost. Data streamer works thought some Ignite node and in case if this the node failed it can't somehow start working with another one. So your application should think about how to track that all data was loaded (wait for completion of

Re: Ignite Persistence: Baseline Topology

Hi, I guess that every data node should have have the same data regions. I checked that in case if you have for example 2 nodes with persistence region in BLT and then start a new node (that isn't the part of BLT) with new region and some cache in this new region then it will produce next

Re: Ignite with Spark Intergration

Hi, In Spark you can use next options: * spark.driver.extraJavaOptions * spark.executor.extraJavaOptions You can path your IGNITE JVM options there like -DIGNITE_QUIET=false. Generally clients will be started on executors during data loading or data reading but you also can start them on

Re: Manage offset of KafkaStreamer

Hi, Generally you can't do it using Ignite KafkaStreamer API because it's not so flexible to configure the KafkaConsumer used inside. Generally it supported in

Re: Ignite on-heap & off-heap caches

Hi, No, heap and off-heap memory are different features. *Heap *memory uses -Xms and -Xmx option to allocate the memory used for different operations and generally can't be used for data storing (in case of you don't set on-heap caching). Java GC will work with current memory. *Off-heap*

Re: Ignite AI learning resources

Hi, I don't think that exists some special resources. You can try to ask your questions on mail lists and read existed documentation: 1)Ignite user list - http://apache-ignite-users.70518.x6.nabble.com/ 2)Ignite developers list - http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Cache data not being stored on server nodes

Hi, Please read about expiration here: https://apacheignite.readme.io/docs/expiry-policies Expiry Policy specifies the amount of time that must pass before an entry is considered expired * In-Memory Mode (data is stored solely in RAM): expired entries are *purged *from RAM completely.

Re: Blocked Worker Thread

Hi Conrad, The reasons can be different. Could you please share the logs? BR, Andrei 11/7/2019 10:35 PM, Conrad Mukai (cmukai) пишет: We are running a cache in a 4 node cluster with atomicityMode set to ATOMIC and have persistence enabled. We repeatedly get a SYSTEM_WORKER_BLOCKED error on

Re: Kubernetes- Failed to retrieve Ignite pods IP addresses.

Hi, Using the search I found that current issue was resolved in next thread: http://apache-ignite-users.70518.x6.nabble.com/Ignite-on-RBAC-enabled-K8s-cluster-td22165.html I guess that you should also read next article: https://apacheignite.readme.io/docs/rbac-authorization BR, Andrei

Re: apache ignite installation fails in kubernetes

Hi, As I know it should be fixed in Apache Ignite 2.7.6. Also as an option you can try to modify your current Docker file as next (master version): https://github.com/apache/ignite/blob/master/docker/apache-ignite/Dockerfile BR, Andrei 11/6/2019 10:33 AM, Gokulnath Chidambaram пишет:

Re: Ignite YARN deployment - how to use TCP IP Discovery?

Hi, I guess you can try collect all IP addresses from all nodes that managed by YARN (where you are going to start Ignite) and add them all to addresses into TcpDiscoveryVmIpFinder part of Ignite configuration. Also you should provide the guarantee that each such hosts will be able to

Re: Not able to start second server node due to authentication failure

Hi, It's correct that SecurityContext is null in your case:     SecurityContext subj = spi.nodeAuth.authenticateNode(node, cred);     if (subj == null) {         // Node has not pass authentication.         LT.warn(log, "Authentication failed [nodeId=" + node.id() +             ", addrs=" +

Re: Get key or cache's updation time?

Hi, I guess that you can use CacheEntry to check that new version of entry is different from previous. Example you can see here: https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheEntry.html In case of you are going to check cache updates then try to use Events

Re: Authorizing thin clients

Hi Kurt, Unfortunately from the box Ignite provide only simple username/password authentication for thin clients. You can read more about it here: https://apacheignite.readme.io/docs/advanced-security BR, Andrei 10/1/2019 3:08 PM, Kurt Semba пишет: Hi all, is there a way to define which

Re: Spark setup

Hi, Ignite 2.7.5 requires the spark of 2.3.X version. You should start the separate Spark and Ignite cluster: https://apacheignite.readme.io/docs/getting-started https://spark.apache.org/docs/2.3.0/spark-standalone.html After that you should provide all required Ignite libraries to you

Re: Authentication

e CREATE command to create new SQL users on a freshly started cluster. How would you execute that using Java code? Would the app need to start the cluster, then use the Ignite JDBC driver to connect to the (PUBLIC) schema of that cluster, then run the CREATE SQL command and then exit? Kurt *

Re: Authentication

Hi, I guess that here Ignite has some documentation gap. Advanced security out of the box will work only with thin connections like webconsole, ODBC/JDBC, etc. To get cluster node authentication you should add GridSecurityProcessor implementation:

Re: Slowness in initial data load using COPY command

Hi, You can try to investigate your cluster state first: 1)First of all try to take a look at your logs and find there long JVM pauses. Also, you can collect the GC log and see the memory usage. It possible that the provided memory for your node isn't enough because it was used for something

Re: Users named "ignite" cannot change their password?

Hi, In addition to Ilia answer, please read next page: https://apacheignite-sql.readme.io/docs/create-user#section-description BR, Andrei 9/4/2019 11:47 AM, Ilya Kasnacheev пишет: Hello! User names are case sensitive and bound by SQL rules, i.e.: ALTER USER "ignite" WITH PASSWORD

Re: Altered sql table (adding new columns) does not reflect in Spark shell

Hi, Yes, I can confirm that this is the issue. I filed next ticket for it: https://issues.apache.org/jira/browse/IGNITE-12159 BR, Andrei 9/7/2019 10:00 PM, Shravya Nethula пишет: Hi, *I created and altered the table using the following queries: * a. CREATE TABLE person (id LONG, name

Re: IgniteCache.invoke deadlock example

Hello, When you use the entry processor then you lock only provided key. So when you tries to work with *other keys* (different from provided one) that are being processed somewhere in other threads then deadlock is possible because other thread can take lock on these *other keys* and wait

Re: Cache expiry policy not deleting records from disk(native persistence)

Hello, I guess that generated WAL will take this disk space. Please read about WAL here: https://apacheignite.readme.io/docs/write-ahead-log Please provide the size of every folder under /opt/ignite/persistence. BR, Andrei 9/6/2019 9:45 PM, Shiva Kumar пишет: Hi all, I have set cache

Re: Ignite ignores cache config when putting entries through near cache

Hi Bartłomiej, Yes, it looks like a bug. Thank you for filing of the JIRA ticket. Possible that http://apache-ignite-developers.2346864.n4.nabble.com is better place to discuss the issues in product. You can start the thread there.

Re: Concurrent threads updating the same cache item

Hi, When you start the compute task then it (and code from there) will be executed on every server (that were chosen) consistently in the single thread. But if you will try to broadcast the same task on several servers then race between different tasks on different servers is possible in

Re: Common table expressions preserved as view.

Hi, At the moment SQL views and temporary tables (CTE) don't fully supported in Ignite. The CTE syntax will just set the sub-query inside your SQL request. BR, Andrei 9/5/2019 4:36 PM, kresimir.horvat пишет: Hi, I noticed that some common table expressions defined in queries are left

Re: Can Ignite transaction manage cached and non-cached data writes?

Hi, By default Ignite will handle only updates that were done using Ignite cache API. I guess that you have already read next article: https://www.gridgain.com/resources/blog/apache-ignite-transactions-architecture-transaction-handling-level-3rd-party BR, Andrei 9/3/2019 2:03 PM, bijunathg

Re: Job Stealing node not stealing jobs

Hi, Some remarks about job stealing SPI: 1)You have some nodes that can proceed the tasks of some compute job. 2)Tasks will be executed in public thread pool by default: https://apacheignite.readme.io/docs/thread-pools#section-public-pool 3)If some node thread pool is busy then some task of

Re: Cache spreading to new nodes

Hi, Could you share the whole reproducer with all configurations and required methods? BR, Andrei 8/12/2019 4:48 PM, Marco Bernagozzi пишет: I have a set of nodes, and I want to be able to set a cache in specific nodes. It works, but whenever I turn on a new node the cache is automatically

Re: TimeoutException not wrapped in CacheException

ogyException - retryReadyFuture().get() and try to rerun transaction CacheException check if getCause is Timeout then try to rerun or rethrow in other cases Thanks. On Fri, Aug 9, 2019 at 5:31 PM Andrei Aleksandrov mailto:aealexsand...@gmail.com>> wrot

Re: TimeoutException not wrapped in CacheException

Hi, It looks strange because even Ignite tests expect that TransactionTimeoutException will be wrapped in CacheException. For example IgniteTxConfigCacheSelfTest:     try (final Transaction tx = ignite.transactions().txStart()) {     assert tx != null;    

Re: Ignite Spark Example Question

Hi, Spark contains several *SaveModes *that will be applied if the table that you are going to use exists: * *Overwrite *- with this option you *will try to re-create* existed table or create new and load data there using IgniteDataStreamer implementation * *Append *- with this option you

Re: IgniteCache.lock behaviour on a key that doesn't exist in the cache (yet)

Hi, I tested your code and looks like it worked fine. However, you also can try to use next method: https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#getAndPutIfAbsent-K-V- In this case you can: 1)Check cache.containsKey(key) 2)If true then prepare the

Re: What happens when a client gets disconnected

Hi, I guess that you should provide the full client and server logs, configuration files and reproducer if it's possible for case when the client node with near cache was able to crush the whole cluster. Looks like it can be the issue here and the best way will be raise the JIRA ticket for

Re: Ignite 2.7.0: Ignite client:: memory leak

Hi Mahesh, Yes, it's a problem related to IGNITE_EXCHANGE_HISTORY_SIZE. Ignite stored the data for the last 1000 exchanges. It generally can be required for the case when the coordinator was changed and new coordinator required to load last exchange history. Exist two problems here:

Re: Can Ignite Kafka connector be able to perform partial update ?

Hi, Unfortunately, Ignite Kafka connector is simple implementation of Kafka connector that use source and sink functions. All data transformation and filtering should be done using Kafka API. I guess that you can try use next functions for your purposes:

Re: Declaring server side CacheEntryListenerin Ignte config

Hi, Could you please provide more details about your case? Generally, for tracking of the cache updates in Ignite you can use Events: https://apacheignite.readme.io/docs/events Continuous Queries: https://apacheignite.readme.io/docs/continuous-queries CacheInterceptor:

Re: Ignite DB Issues

Hi, There are not enough details in your message. 1. I have 10 records of CSV and stored in Ignite DB then ten records will be stored along with new table creation. Now I have removed drop table code from my java code and removed table creation code and running the java code. It is not

Re: about start -f command in ignitevisorcmd

Hi, Looks like this file was moved to https://github.com/gridgain/apache-ignite/blob/ignite-2.7.5/config/visor-cmd/node_startup_by_ssh.sample.ini Here is the ticket: https://issues.apache.org/jira/browse/IGNITE-10036 Please double check your config folder or you can download this file from

Re: Metrics for Ignite

Hi, You can try to use the web console for Ignite. It contains the Monitoring Dashboard with different cache metrics that were calculated during some period of time. Also, it contains different graphics related to cache operation throughput and latency:

Re: AdaptiveLoadBalancingSpi not removing finished tasks from taskTops map

Hi, I tested your case and looks like Ignite have the memory issue here. I filed the ticket for it: https://issues.apache.org/jira/browse/IGNITE-11966 BR, Andrei On 2019/06/03 13:04:49, chris_d wrote: > Hi Andrei,> > > I've attached a zip of the top consumers and the class we're using to> >

Re: Python client: create cache with expiration policy

Hi, Looks like Python API should be improved. I created the ticket on this: https://issues.apache.org/jira/browse/IGNITE-11965 BR, Andrei On 2019/06/03 11:09:29, kulinskyvs wrote: > Hi,> > > Are we talking about the same? I mean, my goal is to create a cache (if not> > yet created) with