date:20210907

RE: Calling Ignite Rest endpoint using secret key

2021-09-07 Thread satyajit.mandal

Hi  Stephen,

Thank  you  very  much  for  your  assistance.

Thanks
Satyajit




Restricted - External
From: Stephen Darlington 
Sent: Tuesday, September 7, 2021 7:48 PM
To: user 
Subject: Re: Calling Ignite Rest endpoint using secret key


CAUTION: This email originated from outside our organisation - 
stephen.darling...@gridgain.com Do not 
click on links, open attachments, or respond unless you recognize the sender 
and can validate the content is safe.
It looks like the implementation doesn't exactly match the comment - should 
probably open a ticket. But, this works:

#!/usr/bin/env python3

import hashlib
import base64
import time
import requests as req

password = 'password'

millis = int(round(time.time() * 1000))
param = f'{millis}:{password}'
hash_pass = hashlib.sha1(param.encode('utf-8')).digest()
str_pass = base64.b64encode(hash_pass).decode('utf-8')

headers = { 'X-Signature' : f'{millis}:{str_pass}' }
resp = req.get('http://localhost:8080/ignite?cmd=version', headers=headers)

print (resp.text)

Basically, you pass in a X-Signature header with a string in the format "x:y".

X = the time in ms
Y = base64 ( sha1 ( "x:secretkey" ) )

Yes, you include the time and use the time as part of the hash. For this (and 
other reasons) I'm not sure that this is a good mechanism. If you want to 
secure it, use TLS, add authentication.

Regards,
Stephen

On 7 Sep 2021, at 12:44, 
satyajit.man...@barclays.com wrote:

Hi  Team,

Can  someone  please  suggest  here.  There  is  no  example  how  to  call a  
rest  url  with  secret  key as per  below  documentation.

setSecretKey(String)
Defines secret key used for client authentication. When provided, client 
request must contain HTTP header X-Signature with the string "[1]:[2]", where 
[1] is timestamp in milliseconds and [2] is the Base64 encoded SHA1 hash of the 
secret key.


Thanks
Satyajit


Restricted - External
From: Mandal, Satyajit: IT (PUN)
Sent: Friday, September 3, 2021 3:41 PM
To: 'user' mailto:user@ignite.apache.org>>
Subject: Calling Ignite Rest endpoint using secret key

Hi  Team,

Can  someone  provide  example  how  we  can  invoke Ignite  Rest  Endpoint  
using  encoded  secret  key  in  header.  There  is  not  much  example .  Any  
sample  will  help which  is  working.


https://ignite.apache.org/docs/latest/restapi





 
   
 
  



setSecretKey(String)
Defines secret key used for client authentication. When provided, client 
request must contain HTTP header X-Signature with the string "[1]:[2]", where 
[1] is timestamp in milliseconds and [2] is the Base64 encoded SHA1 hash of the 
secret key.
Thanks
Satyajit
_
"This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.
For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer
 regarding market commentary from Barclays Sales and/or Trading, who are active 
market participants; 
https://www.investmentbank.barclays.com/disclosures/barclays-global-markets-disclosures.html
 regarding our standard terms for the Investment Bank of Barclays where we 
trade with you in principal-to-principal wholesale markets transactions; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see 
http://publicresearch.barclays.com."
_
If you are incorporated or operating in Australia, please see 
https://www.home.barclays/disclosures/importantapacdisclosures.html

Re: Calling Ignite Rest endpoint using secret key

2021-09-07 Thread Stephen Darlington

It looks like the implementation doesn’t exactly match the comment — should 
probably open a ticket. But, this works:

#!/usr/bin/env python3

import hashlib
import base64
import time
import requests as req

password = 'password'

millis = int(round(time.time() * 1000))
param = f'{millis}:{password}'
hash_pass = hashlib.sha1(param.encode('utf-8')).digest()
str_pass = base64.b64encode(hash_pass).decode('utf-8')

headers = { 'X-Signature' : f'{millis}:{str_pass}' }
resp = req.get('http://localhost:8080/ignite?cmd=version', headers=headers)

print (resp.text)

Basically, you pass in a X-Signature header with a string in the format “x:y”.

X = the time in ms
Y = base64 ( sha1 ( “x:secretkey" ) )

Yes, you include the time and use the time as part of the hash. For this (and 
other reasons) I’m not sure that this is a good mechanism. If you want to 
secure it, use TLS, add authentication.

Regards,
Stephen

> On 7 Sep 2021, at 12:44, satyajit.man...@barclays.com wrote:
> 
> Hi  Team,
>  
> Can  someone  please  suggest  here.  There  is  no  example  how  to  call a 
>  rest  url  with  secret  key as per  below  documentation.
>  
> setSecretKey(String)
> Defines secret key used for client authentication. When provided, client 
> request must contain HTTP header X-Signature with the string "[1]:[2]", where 
> [1] is timestamp in milliseconds and [2] is the Base64 encoded SHA1 hash of 
> the secret key.
>  
>  
> Thanks
> Satyajit
>  
>  
> Restricted - External
> From: Mandal, Satyajit: IT (PUN) 
> Sent: Friday, September 3, 2021 3:41 PM
> To: 'user' 
> Subject: Calling Ignite Rest endpoint using secret key 
>  
> Hi  Team,
>  
> Can  someone  provide  example  how  we  can  invoke Ignite  Rest  Endpoint  
> using  encoded  secret  key  in  header.  There  is  not  much  example .  
> Any  sample  will  help which  is  working.
>  
>  
> https://ignite.apache.org/docs/latest/restapi 
> 
>  
> 
>  
> 
>  
>  
> 
>   class="org.apache.ignite.configuration.ConnectorConfiguration">
> value="these.signatures"/>
>  
>   
> 
>  
>  
>  
> setSecretKey(String)
> Defines secret key used for client authentication. When provided, client 
> request must contain HTTP header X-Signature with the string "[1]:[2]", where 
> [1] is timestamp in milliseconds and [2] is the Base64 encoded SHA1 hash of 
> the secret key.
> Thanks
> 
> Satyajit
> 
> _
> This message is for information purposes only, it is not a recommendation, 
> advice, offer or solicitation to buy or sell a product or service nor an 
> official confirmation of any transaction. It is directed at persons who are 
> professionals and is not intended for retail customer use. Intended for 
> recipient only. This message is subject to the terms at: 
> www.barclays.com/emaildisclaimer .
> 
> For important disclosures, please see: 
> www.barclays.com/salesandtradingdisclaimer 
>  regarding market 
> commentary from Barclays Sales and/or Trading, who are active market 
> participants; 
> https://www.investmentbank.barclays.com/disclosures/barclays-global-markets-disclosures.html
>  
> 
>  regarding our standard terms for the Investment Bank of Barclays where we 
> trade with you in principal-to-principal wholesale markets transactions; and 
> in respect of Barclays Research, including disclosures relating to specific 
> issuers, please see http://publicresearch.barclays.com 
> .  
> _
> If you are incorporated or operating in Australia, please see 
> https://www.home.barclays/disclosures/importantapacdisclosures.html 
> for 
> important disclosure.
> _
> How we use personal information  see our privacy notice 
> https://www.investmentbank.barclays.com/disclosures/personalinformationuse.html
>  
> 
>  
> ___

Re: apache ignite 2.10.0 heap starvation

2021-09-07 Thread Ibrahim Altun

Hi,

totally 400 - 600K reads/writes/updates
12core
64GB RAM
no iowait
10 nodes

On 2021/09/07 12:51:28, Piotr Jagielski  wrote: 
> Hi,
> Can you provide some information on how you use the cluster? How many 
> reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> 
> We observed full GC / CPU load / OOM killer when loading big amount of data 
> (15 mln records, data streamer + allowOverwrite=true). We've seen 200-400k 
> updates per sec on JMX metrics, but load up to 10 on nodes, iowait to 30%. 
> Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, 32GB RAM). 
> Ignite 2.10
> 
> Regards,
> Piotr
> 
> On 2021/09/02 08:36:07, Ibrahim Altun  wrote: 
> > After upgrading from 2.7.1 version to 2.10.0 version ignite nodes facing
> > huge full GC operations after 24-36 hours after node start.
> > 
> > We try to increase heap size but no luck, here is the start configuration
> > for nodes;
> > 
> > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> > -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> > -Dcom.sun.management.jmxremote
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.port=49165
> > -Dcom.sun.management.jmxremote.host=localhost
> > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> > -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
> > -Djava.net.preferIPv4Stack=true"
> > 
> > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> > -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> > -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> > -XX:GCLogFileSize=100M"
> > 
> > here is the 80 hours of GC analyize report:
> > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ==&channel=WEB
> > 
> > do we need more heap size or is there a BUG that we need to be aware?
> > 
> > here is the node configuration:
> > 
> > 
> > http://www.springframework.org/schema/beans";
> >xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> >xsi:schemaLocation="
> > http://www.springframework.org/schema/beans
> > http://www.springframework.org/schema/beans/spring-beans.xsd";>
> >  > class="org.apache.ignite.configuration.IgniteConfiguration">
> > 
> > 
> >  > value="/etc/apache-ignite/ignite-log4j2.xml"/>
> > 
> > 
> > 
> >  > class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >  > value="/tmp/temp_ignite_libs"/>
> > 
> > 
> > 
> > file://freq=5000@localhost/usr/share/apache-ignite/libs/segmentify/
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >  > class="org.apache.ignite.configuration.CacheConfiguration">
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >  > class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >  > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > 
> >  > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > 
> >  > value="#{ 2L * 1024 * 1024 * 1024}"/>
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > -- 
> > İbrahim Halil AltunSenior Software Engineer+90
> > 536 3327510 • segmentify.com → UK • Germany •
> > Turkey 
> > 
> > 
>

Re: apache ignite 2.10.0 heap starvation

2021-09-07 Thread Piotr Jagielski

Hi,
Can you provide some information on how you use the cluster? How many 
reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?

We observed full GC / CPU load / OOM killer when loading big amount of data (15 
mln records, data streamer + allowOverwrite=true). We've seen 200-400k updates 
per sec on JMX metrics, but load up to 10 on nodes, iowait to 30%. Our cluster 
is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, 32GB RAM). Ignite 2.10

Regards,
Piotr

On 2021/09/02 08:36:07, Ibrahim Altun  wrote: 
> After upgrading from 2.7.1 version to 2.10.0 version ignite nodes facing
> huge full GC operations after 24-36 hours after node start.
> 
> We try to increase heap size but no luck, here is the start configuration
> for nodes;
> 
> JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.port=49165
> -Dcom.sun.management.jmxremote.host=localhost
> -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
> -Djava.net.preferIPv4Stack=true"
> 
> JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> -XX:GCLogFileSize=100M"
> 
> here is the 80 hours of GC analyize report:
> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ==&channel=WEB
> 
> do we need more heap size or is there a BUG that we need to be aware?
> 
> here is the node configuration:
> 
> 
> http://www.springframework.org/schema/beans";
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd";>
>  class="org.apache.ignite.configuration.IgniteConfiguration">
> 
> 
>  value="/etc/apache-ignite/ignite-log4j2.xml"/>
> 
> 
> 
>  class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  value="/tmp/temp_ignite_libs"/>
> 
> 
> 
> file://freq=5000@localhost/usr/share/apache-ignite/libs/segmentify/
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
> 
>  class="org.apache.ignite.configuration.DataRegionConfiguration">
> 
>  value="#{ 2L * 1024 * 1024 * 1024}"/>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> İbrahim Halil AltunSenior Software Engineer+90
> 536 3327510 • segmentify.com → UK • Germany •
> Turkey 
> 
>

2021-09-07 Thread Ibrahim Altun

please subscribe me

-- 
İbrahim Halil AltunSenior Software Engineer+90
536 3327510 • segmentify.com → UK • Germany •
Turkey

RE: Calling Ignite Rest endpoint using secret key

2021-09-07 Thread satyajit.mandal

Hi  Team,

Can  someone  please  suggest  here.  There  is  no  example  how  to  call a  
rest  url  with  secret  key as per  below  documentation.

setSecretKey(String)
Defines secret key used for client authentication. When provided, client 
request must contain HTTP header X-Signature with the string "[1]:[2]", where 
[1] is timestamp in milliseconds and [2] is the Base64 encoded SHA1 hash of the 
secret key.


Thanks
Satyajit



Restricted - External
From: Mandal, Satyajit: IT (PUN)
Sent: Friday, September 3, 2021 3:41 PM
To: 'user' 
Subject: Calling Ignite Rest endpoint using secret key

Hi  Team,

Can  someone  provide  example  how  we  can  invoke Ignite  Rest  Endpoint  
using  encoded  secret  key  in  header.  There  is  not  much  example .  Any  
sample  will  help which  is  working.



https://ignite.apache.org/docs/latest/restapi







 
   
 
  



setSecretKey(String)
Defines secret key used for client authentication. When provided, client 
request must contain HTTP header X-Signature with the string "[1]:[2]", where 
[1] is timestamp in milliseconds and [2] is the Base64 encoded SHA1 hash of the 
secret key.

Thanks

Satyajit

_
This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; 
https://www.investmentbank.barclays.com/disclosures/barclays-global-markets-disclosures.html
 regarding our standard terms for the Investment Bank of Barclays where we 
trade with you in principal-to-principal wholesale markets transactions; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.  
_
If you are incorporated or operating in Australia, please see 
https://www.home.barclays/disclosures/importantapacdisclosures.html for 
important disclosure.
_
How we use personal information  see our privacy notice 
https://www.investmentbank.barclays.com/disclosures/personalinformationuse.html 
_

Re: Ignite CheckpointReadLock /Long running cache futures

2021-09-07 Thread Ilya Kazakov

Hello Mike. According to your description, there was a hanging PME. If you
need more detailed analysis, could you share your logs and thread dumps?

---
Ilya

пн, 6 сент. 2021 г. в 22:21, Mike Wiesenberg :

> Using Ignite 2.10.0
>
> We had a frustrating series of issues with Ignite the other day. We're
> using a 4-node cluster with 1 backup per table and cacheMode set to
> Partitioned, and write behind enabled. We have a client that inserts data
> into caches and another client that listens for new data in those caches.
> (Apologies I can't paste logs or configuration due to firm policy)
>
> What happened:
>
> 1. We observed that our insertion client was not working after startup, it
> logged every 20 seconds that 'Still awaiting for initial partition map
> exchange.' This continued until we restarted the node it was trying to
> connect to, at which point the client connected to another node and the
> warning stopped.
>
>  Possible Bug #1 - why didn't it automatically try a different node, or if
> it would have that same issue connecting to any node, why couldn't the
> cluster print an error and function anyhow?
>
> 2. After rebooting bad node #1, the insertion client still didn't work, it
> then started printing totally different warnings about 'First 10 long
> running cache futures [total=1]', whatever that means, and then printed the
> ID of a node. We killed that referenced node, and then everything started
> working.
>
>  Again, why didn't the client switch to a good node automatically(or is
> there a way to configure such failover capability that I don't know about)?
>
> 3. In terms of root cause, it seems bad node #1 had a 'blocked
> system-critical thread' which according to the stack trace was blocked at
> CheckpointReadWriteLock.java line 69. Is there a way to automatically
> recover from this or handle this more gracefully? If not I will probably
> disable WAL (which I understand will disable checkpointing).
>
>  Possible Bug #2 - why couldn't it recover from this lock if restarting
> fixed it?
>
> Regards, and thanks in advance, for any advice!
>

RE: Calling Ignite Rest endpoint using secret key

Re: Calling Ignite Rest endpoint using secret key

Re: apache ignite 2.10.0 heap starvation

Re: apache ignite 2.10.0 heap starvation

Subscribe

RE: Calling Ignite Rest endpoint using secret key

Re: Ignite CheckpointReadLock /Long running cache futures

7 matches

Site Navigation

Mail list logo

Footer information