Re: Unsubscribe

[Jessie Lin]

Thank you Denis. I've seen great work from Ignite. Keep up!

On Wed, Sep 20, 2017 at 8:33 AM, Denis Mekhanikov 
wrote:

> Hi Jessie.
>
> Sorry to see you go.
> To unsubscribe from the user mailing list send a letter to
> user-unsubscr...@ignite.apache.org with a word "Unsubscribe" without
> quotes as a topic.
> If you have a mailing client, follow an unsubscribe link here:
> https://ignite.apache.org/community/resources.html#mail-lists
>
> Denis
>
> ср, 20 сент. 2017 г. в 18:22, Jessie Lin :
>
>>
>> Please unsubscribe me from the mailing list.
>> I tried several times but still receive emails.
>> What's right way to do that? thank you.
>>
>> Jessie
>>
>>
>> On Fri, Sep 8, 2017 at 4:47 AM, 谢宁  wrote:
>>
>>>
>>>
>>>
>>> -- Original --
>>> *From:* waterg 
>>> *Date:* Fri,Sep 8,2017 0:03 PM
>>> *To:* user 
>>> *Subject:* Re: unsubscribe
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>

Unsubscribe

[Jessie Lin]

Please unsubscribe me from the mailing list.
I tried several times but still receive emails.
What's right way to do that? thank you.

Jessie


On Fri, Sep 8, 2017 at 4:47 AM, 谢宁  wrote:

>
>
>
> -- Original --
> *From:* waterg 
> *Date:* Fri,Sep 8,2017 0:03 PM
> *To:* user 
> *Subject:* Re: unsubscribe
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Unsubscribe

[Jessie Lin]

Please unsubscribe

Re: DataStreamer operation failed

[Jessie Lin]

Hello Pranas, We had similar issues when having client node and server.
If you find out how to fix it, would appreciate if you could post the
solution here.

Jessie

On Mon, Aug 28, 2017 at 4:57 PM, Pranas Baliuka 
wrote:

> Thanks Konstantin in looking at it,
>
> The issue was what server node and client node serialization was not
> compatible. Moved to
> and using single server
> type node for further testing.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/DataStreamer-operation-failed-tp16439p16465.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Streaming test

[Jessie Lin]

Hi Val, thank you getting back.  The source type will be different, and the
client will deal with the difference in a preprocess that's unique to that
data source. Once that's done, it'll be ready for a standard
transformation. So we can encapsulate it in a single service or callable.

Thanks for explaining the difference between client and servers. It
clarifies things and make sense.
We wanted to build different type of clients to accommodate the differences
in the data sources.

Does this make sense? what kind of solutions would you suggest?

Jessie



On Fri, Aug 18, 2017 at 3:34 PM, vkulichenko 
wrote:

> If data sources are different, then transformations are different as well,
> no? If so, how are you going to encapsulate them in a single service? I
> would really appreciate a concrete example, it might make this much easier.
> I'm probably missing something now.
>
> Difference between clients and servers is the same is in other systems.
> Servers store data, run services and computations, etc. while clients are
> only used to interact with the cluster.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16310.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Streaming test

[Jessie Lin]

We'd like to take either input from streaming or a cache to do some
transformation and insert data into another cache.
Because the input could be different types, I thought I could encapsulate
the transformation and just let a client to call it.

The difference between client and server in Ignite isn't very clear to me,
to be honest :)
One thing I can think of is client doesn't hold data. But I don't know the
difference in terms of computation.

Jessie


On Thu, Aug 17, 2017 at 3:56 PM, vkulichenko 
wrote:

> What does business logic do exactly? Is it supposed to be executed on
> server
> or client side?
>
> Can you give an example?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16270.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Streaming test

[Jessie Lin]

Val, I'd like to build a piece of logic ( could be service, callable, etc)
and encapsulate business logic there. And build different clients to for
different type of sources, for example file, streaming, jdbc, etc, the
client will invoke the business logic by calling the service or callables.

What architecture/components would you recommend to use in this type of
situation?
Thank you very much!

Jessie

On Tue, Aug 15, 2017 at 5:28 PM, vkulichenko 
wrote:

> Can you describe the use case and what you want to achieve, step by step?
> I'll try to suggest a solution.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16213.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Streaming test

[Jessie Lin]

Btw, I tried two ways to get service proxy on a client node, the first one
worked fine, but the one with Stream Visitor failed.
1) I stream the file and put them into a cache, and then loop through the
cache entries and call Service
2) I stream the file and define a stream visitor, in which I get the
key(String) and value(a defined class) from the Map and call the
service. The service interface is defined as String, the defined class.

Kinda interesting... Thank you for helping out!


On Tue, Aug 15, 2017 at 10:07 AM, Jessie Lin 
wrote:

> The class gets a service proxy and invokes the service.
> I thought service could be invoked on client as well...Looks like it's not
> designed to be used that way.
>
> If I wrap the Service in a Callable, can I call it from a client node?
> The client node would stream from a local file and call the Callable, I
> think?
> Does that sound right?
>
> Jessie
>
> On Mon, Aug 14, 2017 at 5:30 PM, vkulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> What is this class used for? I'm a bit confused - service is usually
>> invoked
>> on the server, so it will obviously need all classes it works with. Am I
>> missing something?
>>
>> Can you show the whole trace?
>>
>> -Val
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Streaming-test-tp14039p16189.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: Streaming test

[Jessie Lin]

The class gets a service proxy and invokes the service.
I thought service could be invoked on client as well...Looks like it's not
designed to be used that way.

If I wrap the Service in a Callable, can I call it from a client node?
The client node would stream from a local file and call the Callable, I
think?
Does that sound right?

Jessie

On Mon, Aug 14, 2017 at 5:30 PM, vkulichenko 
wrote:

> What is this class used for? I'm a bit confused - service is usually
> invoked
> on the server, so it will obviously need all classes it works with. Am I
> missing something?
>
> Can you show the whole trace?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16189.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Use SLF4J in Ignite

[Jessie Lin]

Hi Dmitry, thanks for getting back. I verified IGNITE_HOME env variable.
It's pointed to the same folder...
Btw, I did notice that it was set as a User env variable instead of system
env variable in Windows.
I changed in to System env variable and will run the test again.

On Tue, Aug 15, 2017 at 3:00 AM, dkarachentsev 
wrote:

> Hi,
>
> Check where IGNITE_HOME env variable points to, Ignite may load libraries
> out of there.
>
> Thanks!
> -Dmitry.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Use-SLF4J-in-Ignite-tp16179p16199.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Use SLF4J in Ignite

[Jessie Lin]

Yes, the ignite-slf4j folder is copied over

[image: Inline image 1]

On Mon, Aug 14, 2017 at 1:10 PM, ignite_user2016 
wrote:

> It looks like ignite version of slf4j is not in the classpath.
>
> Did you copy the ignite slf4j jars into server lib directory ?
>
> On Mon, Aug 14, 2017 at 1:55 PM, waterg [via Apache Ignite Users] <[hidden
> email] > wrote:
>
>> Hello there,
>>
>> I'm having following error when starting Ignite server node
>>
>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further detail
>> s.
>>
>> In maven pom file, I added
>> 
>> org.apache.ignite
>> ignite-slf4j
>> ${ignite.version}
>> 
>> In XML config, I added:
>> 
>> 
>> 
>> I've also moved 'optional/ignite-slf4j' folder to
>> 'libs' folder before running 'ignite.{sh|bat}' script, according to
>> README here https://github.com/apache/ignite/tree/master/modules/slf4j
>>
>> What should I do differently?
>>
>> Thank you.
>> Jessie
>>
>> --
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-ignite-users.70518.x6.nabble.com/Use-SLF4J-in-
>> Ignite-tp16179.html
>> To start a new topic under Apache Ignite Users, email [hidden email]
>> 
>> To unsubscribe from Apache Ignite Users, click here.
>> NAML
>> 
>>
>
>
>
> --
> Rishi Yagnik
>
> --
> View this message in context: Re: Use SLF4J in Ignite
> 
>
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>

Re: Streaming test

[Jessie Lin]

Val, thanks for pointing it out. Now I call AtomicLong Function from
service#execute() and it's working. Thank you very much!

Jessie

On Thu, Aug 10, 2017 at 3:08 PM, vkulichenko 
wrote:

> Jessie,
>
> You still call atomicLong() method from Service#init(). As I already
> mentioned, this is causing the startup hang. You should move
> IgniteAtomicLong creation out of init() method to avoid it.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16113.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Streaming test

[Jessie Lin]

Val, please see thread print attached.
This is take after a server is run by "bin\ignite.bat
config\ignite-writebehind.xml" and the service initialization didn't
complete.
Thank you very much for helping out!

"srvc-deploy-#33%null%" #59 prio=5 os_prio=0 tid=0x577b8000
nid=0x1ef8 waiting on condition [0x608fe000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xc09fbdc8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.apache.ignite.internal.util.IgniteUtils.await(IgniteUtils.java:7419)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.awaitInitialization(DataStructuresProcessor.java:1112)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.atomicLong(DataStructuresProcessor.java:517)
at
org.apache.ignite.internal.IgniteKernal.atomicLong(IgniteKernal.java:3436)
at com.sample.SampleServiceImpl.init(SampleServiceImpl.java:63)

Jessie

On Wed, Aug 9, 2017 at 2:29 PM, vkulichenko 
wrote:

> I can't reproduce it, your project works fine for me. Can you attach thread
> dumps?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16088.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
12484:
2017-08-10 12:50:26
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.91-b14 mixed mode):

"sys-#56%null%" #82 prio=5 os_prio=0 tid=0x58ad8000 nid=0x33dc waiting 
on condition [0x5b49f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xc0277e70> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sys-#55%null%" #81 prio=5 os_prio=0 tid=0x58ad7800 nid=0x1c50 waiting 
on condition [0x5ab9e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xc0277e70> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sys-#54%null%" #80 prio=5 os_prio=0 tid=0x58ad6800 nid=0x2a40 waiting 
on condition [0x5b5ff000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xc0277e70> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sys-#53%null%" #79 prio=5 os_prio=0 tid=0x586f1000 nid=0x3f3c waiting 
on condition [0x5a84e000]
   java.lang.Thread.State: TIMED_WAITING (park

Re: Streaming test

[Jessie Lin]

Val, I think the ignite.atomicLong() function never returned and just hang
there. If you run the test project I attached it should be reproducable.

I don't expect to have memory issue here. As the service init doesn't do
much else. And I only have one server node and one client node.

Jessie

On Tue, Aug 8, 2017 at 5:36 PM, vkulichenko 
wrote:

> Jessie,
>
> What do you mean by "stuck"? Did you check thread dumps? Is it possible you
> have memory/GC issues?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16066.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Streaming test

[Jessie Lin]

Val, I replace atomicSequence with atomicLong and used the method below.
I need to specify an initial value, start, according to requirement.

return ignite.atomicLong(
"id", //  name.
start,   // Initial value
true // Create if it does not exist.
);

But the process got stuck as well.
Shall I do something differently? Appreciate your advice.

Best,
Jessie

On Tue, Jul 25, 2017 at 7:55 PM, Jessie Lin 
wrote:

> Thank you Val. I'll give it a try.
>
> On Tue, Jul 25, 2017 at 5:16 PM, vkulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Jessie,
>>
>> atomicLong() method will do this for you automatically. It either creates
>> new instance or returns existing one. There is no need for separate
>> initialization step.
>>
>> -Val
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Streaming-test-tp14039p15646.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: Streaming test

[Jessie Lin]

Thank you Val. I'll give it a try.

On Tue, Jul 25, 2017 at 5:16 PM, vkulichenko 
wrote:

> Jessie,
>
> atomicLong() method will do this for you automatically. It either creates
> new instance or returns existing one. There is no need for separate
> initialization step.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p15646.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Streaming test

[Jessie Lin]

Val, thank you for the reply.

Unfortunately I don't have an existing AtomicLong or AtomicSequence to work
with... What if I need to initiate one, where shall I initiate it? We have
a small cluster, sometime I only have one node for dev/test. It's a bit
surprising why initiate an AtomicSequence would need to wait so long,
because there's no other node to wait for...



Jessie

On Mon, Jul 24, 2017 at 4:33 PM, vkulichenko 
wrote:

> Jessie,
>
> Ignite#atomicLong method will return existing instance if it is already
> initialized, so you actually don't need to manage this manually.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p15525.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Streaming test

[Jessie Lin]

Thank you for running the test.
If I need to initialize Atomic Seq, where shall I do it?

Jessie




On Fri, Jul 21, 2017 at 12:37 PM, vkulichenko  wrote:

> I ran the test. Atomic sequence initialization should be moved out of
> init()
> because it is called synchronously with node start, and they wait for each
> other.
>
> After I do this, everything works fine. Although CSV file has only 348 and
> it gets loaded multiple times in a loop.
>
> Is there anything else I need to do to reproduce the hang? In addition, if
> you do have the hang, take thread dumps from all nodes and attach them
> here.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p15263.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Ignite 2.0 off-heap memory

[Jessie Lin]

Thank you for confirming and opening the ticket Mikhail.

Jessie

On Fri, Jun 9, 2017 at 6:58 AM, Michael Cherkasov <
michael.cherka...@gmail.com> wrote:

> Hi Jessie,
>
> With your configuration Ignites 2.0 stores data in Off Heap memory, but
> visor shows
> wrong statistics, I created a ticket for this issue:
>
> https://issues.apache.org/jira/browse/IGNITE-5461
>
> Thanks,
> Mikhail.
>
>
> 2017-06-09 1:48 GMT+03:00 waterg :
>
>> Hello,
>>
>> has anyone used visor to check the memory usage in 2.0?
>> I set up a custom region and used the custom region in the cache
>> configuration.
>> But when I used visor, it shows that all the entries are on heap. Should I
>> do something different?
>>
>> memory config:
>>  
>> > class="org.apache.ignite.configuration.MemoryConfiguration">
>>
>> 
>> 
>> > class="org.apache.ignite.configuration.MemoryPolicyConfiguration">
>> 
>>
>> 
>>
>> 
>>
>> > />
>> 
>> 
>> 
>> 
>> 
>>
>> part of cache config
>> > class="org.apache.ignite.configuration.CacheConfiguration">
>>
>> 
>> > value="customRegion"/>
>> 
>> 
>>
>> display of cache -a in visor
>>
>> Nodes for: sampeCache(@c1)
>> +===
>> ===
>> +
>> |  Node ID8(@), IP   | CPUs | Heap Used | CPU Load |   Up Time
>> |
>> Size | Hi/Mi/Rd/Wr   |
>> +===
>> ===
>> +
>> | 8B41D11E(@n0), xxx.xxx.xxx.118 | 8| 5.27 %| 0.33 %   |
>> 00:01:26:659 | Total: 2000  | Hi: 0
>> |
>> ||  |   |  |
>> |
>> Heap: 2000 | Mi: 0 |
>> ||  |   |  |
>> |
>> Off-Heap: 0| Rd: 0 |
>> ||  |   |  |
>> |
>> Off-Heap Memory: 0 | Wr: 0 |
>> +---
>> ---
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Ignite-2-0-off-heap-memory-tp13535.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: Docker cluster set up

[Jessie Lin]

Thank you Alex!

On Wed, May 10, 2017 at 1:54 AM, afedotov 
wrote:

> Hi.
> Did I get it right that containers reside on different hosts? If it's so
> then containers won't see each other out of box. You probably need to start
> them via docker overlay network or use Swarm.
>
> Kind regards,
> Alex
>
> 9 мая 2017 г. 12:29 AM пользователь "waterg [via Apache Ignite Users]" 
> <[hidden
> email] > написал:
>
> All,
>
> I'm trying to set up a cluster using docker. The network setting seems to
> have some problem.
> Could you please take a look and advise?
> I have two hosts and each will run a container. Static ip address and
> basic address resolver are used.
> This is part of configuration on one container.
>
> 
> 
>
> 
>
>
> 
> 
> 
>
>
> host1 public ip
>
> host2 public ip
> 
> 
> 
> 
> 
> 
>
>  
>  
>  
>  
>  
>   value="container 1 private ip"/>
>  
>  
>  
>  
>
> The other container has the same configuration under
> TcpDiscoveryVmIpFinder, but the BasicAddressResolver has ip address for
> host2/container2.
>
> when I run the container, I use command "sudo docker run -p 47100:47100 -p
> 47500:47500 imagename"
>
> Here's the result of docker ps on two hosts.
>
> $ sudo docker ps
> CONTAINER IDIMAGE   COMMAND
>  CREATED STATUS
> PORTS
>  NAMES
> 3e1e86d18b43imagename   "/bin/sh -c $IGNITE_H"   27 seconds ago
>Up 26seconds   
> 0.0.0.0:47100->47100/tcp,
> 11211/tcp, 49112/tcp, 0.0.0.0:47500->47500/tcp   nostalgic_
> noyce
>
>
> $ sudo docker ps
> CONTAINER IDIMAGE   COMMAND
>  CREATED STATUS  PORTS
>  NAMES
> 5bbd182dca5cimagename   "/bin/sh -c $IGNITE_H"   19 minutes ago
>Up 19 minutes   0.0.0.0:47100->47100/tcp, 11211/tcp, 49112/tcp,
> 0.0.0.0:47500->47500/tcp   romantic_galileo
>
>
> Does the settings look right? Thank you!
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Docker-cluste
> r-set-up-tp12549.html
> To start a new topic under Apache Ignite Users, email [hidden email]
> 
> To unsubscribe from Apache Ignite Users, click here.
> NAML
> 
>
>
>
> --
> View this message in context: Re: Docker cluster set up
> 
>
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>

Re: Query search performance and scalability

[Jessie Lin]

Hi Alex, the ping between the two nodes are mostly within 0.1-0.2 ms. I did
see once in a while couple ms. Is this the normal range you'd expect?

Jessie



On Thu, May 4, 2017 at 11:37 PM, afedotov 
wrote:

> Hi Jessie.
>
> What is the ping between physical nodes?
> As well, please try adding two more nodes and share the results.
> I think in your case the query is pretty fast initially, so adding more
> nodes just add to the total time because of more network messages being
> sent. In real life scenarios when executing a query takes several times
> more than network overhead you will gain from scaling the cluster.
>
>
>
> Kind regards,
> Alex
>
>
> 5 мая 2017 г. 8:04 AM пользователь "waterg [via Apache Ignite Users]" <[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=12448&i=0>> написал:
>
> Here's the cache configuration.
> We were expecting to see positive scalability.
> What would be reasons to prevent from that?
>
> Btw, the cache has 300K entries in total.
> Thank you!
>
> 
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
>  factory-method="factoryOf">
>  value="com.sample.store.SampleStore">
> 
> 
> 
> 
> 
>  value="java.lang.String">
>  value="com.sample.model.SampleModel">
> 
> 
> 
>  value="java.lang.String">
>  value="java.util.String">
> 
> 
> 
> 
> 
> 
>         
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
> 
>
>
> On Thu, May 4, 2017 at 9:35 AM, Jessie Lin <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=12447&i=0>> wrote:
>
>> Alex, thanks for reply.
>> The query is fairly simple, colA =?
>> colA is String.
>> For 1 server 1 client scenario, they are the same server.
>> For 2 server 1 client scenario, one server and client are on one server,
>> the other server node is on another server.
>> Working on a simplified code to reproduce the problem to share with you
>>
>> Jessie
>>
>>
>> On Wed, May 3, 2017 at 11:14 PM, afedotov <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=12447&i=1>> wrote:
>>
>>> That should be because of the overhead of network roundtrips between the
>>> client and the servers. Are servers and client located on the same physical
>>> machine?
>>> Please provide the query and cache config.
>>>
>>> Kind regards,
>>> Alex
>>>
>>> 4 мая 2017 г. 3:36 AM пользователь "waterg [via Apache Ignite Users]" 
>>> <[hidden
>>> email] <http:///user/SendEmail.jtp?type=node&node=12407&i=0>> написал:
>>>
>>> I have a program that issues SQLQuery on colA, colA has a single column
>>> index.
>>> The cache has about 360K entries, and 298K unique values in colA.
>>> I ran the query 1M times with different values.
>>>
>>> If the cache only has cache key, here's the results I ran with three
>>> senario
>>> With 1 server node: average search time is 35K nanosecs
>>> With 1 server – 1 client: average search time is 128K nanosecs
>>> With 2 server – 1 client: average search time is 276K nanosecs
>>>
>>> If the cache has Affinity Key, here's the results I ran with three
>>> senario
>>> With 1 server node: average search time is 59K nanosecs
>>> With 1 server – 1 client: average search time is 148K nanosecs
>>> With 2 server – 1 client: average search time is 287K nanosecs
>>>
>>> Could anyone advise why the search speed degrades as more nodes are
>>> added?
>>> Thank you!
>>>
>>> --
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-ignite-users.70518.x6.nabble.com/Que

Re: Query search performance and scalability

[Jessie Lin]

Here's the cache configuration.
We were expecting to see positive scalability.
What would be reasons to prevent from that?

Btw, the cache has 300K entries in total.
Thank you!




















































On Thu, May 4, 2017 at 9:35 AM, Jessie Lin 
wrote:

> Alex, thanks for reply.
> The query is fairly simple, colA =?
> colA is String.
> For 1 server 1 client scenario, they are the same server.
> For 2 server 1 client scenario, one server and client are on one server,
> the other server node is on another server.
> Working on a simplified code to reproduce the problem to share with you
>
> Jessie
>
>
> On Wed, May 3, 2017 at 11:14 PM, afedotov 
> wrote:
>
>> That should be because of the overhead of network roundtrips between the
>> client and the servers. Are servers and client located on the same physical
>> machine?
>> Please provide the query and cache config.
>>
>> Kind regards,
>> Alex
>>
>> 4 мая 2017 г. 3:36 AM пользователь "waterg [via Apache Ignite Users]" 
>> <[hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=12407&i=0>> написал:
>>
>> I have a program that issues SQLQuery on colA, colA has a single column
>> index.
>> The cache has about 360K entries, and 298K unique values in colA.
>> I ran the query 1M times with different values.
>>
>> If the cache only has cache key, here's the results I ran with three
>> senario
>> With 1 server node: average search time is 35K nanosecs
>> With 1 server – 1 client: average search time is 128K nanosecs
>> With 2 server – 1 client: average search time is 276K nanosecs
>>
>> If the cache has Affinity Key, here's the results I ran with three
>> senario
>> With 1 server node: average search time is 59K nanosecs
>> With 1 server – 1 client: average search time is 148K nanosecs
>> With 2 server – 1 client: average search time is 287K nanosecs
>>
>> Could anyone advise why the search speed degrades as more nodes are
>> added?
>> Thank you!
>>
>> --
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-ignite-users.70518.x6.nabble.com/Query-search-
>> performance-and-scalability-tp12401.html
>> To start a new topic under Apache Ignite Users, email [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=12407&i=1>
>> To unsubscribe from Apache Ignite Users, click here.
>> NAML
>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>> --
>> View this message in context: Re: Query search performance and
>> scalability
>> <http://apache-ignite-users.70518.x6.nabble.com/Query-search-performance-and-scalability-tp12401p12407.html>
>>
>> Sent from the Apache Ignite Users mailing list archive
>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>>
>
>

Re: Query search performance and scalability

[Jessie Lin]

Alex, thanks for reply.
The query is fairly simple, colA =?
colA is String.
For 1 server 1 client scenario, they are the same server.
For 2 server 1 client scenario, one server and client are on one server,
the other server node is on another server.
Working on a simplified code to reproduce the problem to share with you

Jessie


On Wed, May 3, 2017 at 11:14 PM, afedotov 
wrote:

> That should be because of the overhead of network roundtrips between the
> client and the servers. Are servers and client located on the same physical
> machine?
> Please provide the query and cache config.
>
> Kind regards,
> Alex
>
> 4 мая 2017 г. 3:36 AM пользователь "waterg [via Apache Ignite Users]" <[hidden
> email] > написал:
>
> I have a program that issues SQLQuery on colA, colA has a single column
> index.
> The cache has about 360K entries, and 298K unique values in colA.
> I ran the query 1M times with different values.
>
> If the cache only has cache key, here's the results I ran with three
> senario
> With 1 server node: average search time is 35K nanosecs
> With 1 server – 1 client: average search time is 128K nanosecs
> With 2 server – 1 client: average search time is 276K nanosecs
>
> If the cache has Affinity Key, here's the results I ran with three senario
> With 1 server node: average search time is 59K nanosecs
> With 1 server – 1 client: average search time is 148K nanosecs
> With 2 server – 1 client: average search time is 287K nanosecs
>
> Could anyone advise why the search speed degrades as more nodes are added?
> Thank you!
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Query-search-
> performance-and-scalability-tp12401.html
> To start a new topic under Apache Ignite Users, email [hidden email]
> 
> To unsubscribe from Apache Ignite Users, click here.
> NAML
> 
>
>
>
> --
> View this message in context: Re: Query search performance and scalability
> 
>
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>

Re: CacheStore's Performance Drops Dramatically - Why?

[Jessie Lin]

I thought flushsize could be set as several times higher than the batch
size is that in a cluster, data nodes would flush in parallel. For example
there's a cluster with 10 nodes, and flushSize is 10240, thread count = 2,
batch size = 512. Then each node would flush out in 2 thread, and each
thread flushes out in batch of 512.

Could someone confirms or clarify the understanding? Thank you!

On Wed, May 3, 2017 at 12:16 AM, Matt  wrote:

> In fact, I don't see why you would need both batchSize and flushSize. If
> I got it right, only the min of them would be used by Ignite to know when
> to flush, why do we have both in the first place?
>
> In case they're both necessary for a reason I'm not seeing, I still wonder
> if the default values should be batchSize > flushSize as I think or not.
>
> On Wed, May 3, 2017 at 3:26 AM, Matt  wrote:
>
>> I'm writing to confirm I managed to fix my problem by fine tuning the
>> config params for the write behind cache until the performance was fine. I
>> still see single element inserts from time to time, but just a few of them
>> every now and then not like before. You should definitely avoid synchronous
>> single elements insertions, I hope that changes in future versions.
>>
>> Regarding writeBehindBatchSize and writeBehindFlushSize, I don't see the
>> point of setting both values when batchSize < flushSize (default values are
>> 512 and 10240 respectively). If I'm not wrong, the cache is flushed
>> whenever the its size is equal to min(batchSize, flushSize). Since
>> batchSize is less than flushSize, flushSize is never really used and the
>> size of the flush is controlled by the size of the cache itself only.
>>
>> That is how it works by default, on the other hand if we swap their
>> values (ie, batchSize=10240 and flushSize=512) the behavior would be the
>> same (Ignite would call writeAll() with 512 elements each time), but the
>> number of elements flushed would be controlled by the correct variable (ie,
>> flushSize).
>>
>> Were the default values supposed to be the other way around or am I
>> missing something?
>>
>> On Tue, May 2, 2017 at 9:13 PM, Denis Magda  wrote:
>>
>>> Matt,
>>>
>>> Cross-posting to the dev list.
>>>
>>> Yes, Ignite switches to the synchronous mode once the buffer is
>>> exhausted. However, I do agree that it would be a right solution to flush
>>> multiple entries rather than one in the synchronous mode. *Igniters*, I was
>>> sure we had a ticket for that optimization but unable to find it.  Does
>>> anybody know the ticket name/number?
>>>
>>> To omit the performance degradation you have to tweak the following
>>> parameters so that the write-behind store can keep up with you updates:
>>> - setWriteBehindFlushThreadCount
>>> - setWriteBehindFlushFrequency
>>> - setWriteBehindBatchSize
>>> - setWriteBehindFlushSize
>>>
>>> Usually it helped all the times to Apache Ignite users.
>>>
>>> > QUESTION 2
>>> >
>>> > I've read on the docs that using ATOMIC mode (default mode) is better
>>> for performance, but I'm not getting why. If I'm not wrong using
>>> TRANSACTIONAL mode would cause the CacheStore to reuse connections (not
>>> call openConnection(autocommit=true) on each writeAll()).
>>> >
>>> > Shouldn't it be better to use transactional mode?
>>>
>>> Transactional mode enables 2 phase commit protocol:
>>> https://apacheignite.readme.io/docs/transactions#two-phase-commit-2pc
>>>
>>> This is why atomic operations are swifter in general.
>>>
>>> —
>>> Denis
>>>
>>> > On May 2, 2017, at 10:40 AM, Matt  wrote:
>>> >
>>> > No, only with inserts, I haven't tried removing at this rate yet but
>>> it may have the same problem.
>>> >
>>> > I'm debugging Ignite internal code and I may be onto something. The
>>> thing is Ignite has a cacheMaxSize (aka, WriteBehindFlushSize) and
>>> cacheCriticalSize (which by default is cacheMaxSize*1.5). When the cache
>>> reaches that size Ignite starts writing elements SYNCHRONOUSLY, as you can
>>> see in [1].
>>> >
>>> > I think this makes things worse since only one single value is flushed
>>> at a time, it becomes much slower forcing Ignite to do more synchronous
>>> writes.
>>> >
>>> > Anyway, I'm still not sure why the cache reaches that level when the
>>&

Re: CacheStore's Performance Drops Dramatically - Why?

[Jessie Lin]

I noticed that behavior when any cache.remove operation is involved. I keep
putting stuff in cache seems to be working properly.

Do you use remove operation?

On Tue, May 2, 2017 at 9:57 AM, Matt  wrote:

> I'm stuck with that. No matter what config I use (flush size, write
> threads, etc) this is the behavior I always get. It's as if Ignite internal
> buffer is full and it's trying to write and get rid of the oldest (one)
> element only.
>
> Any idea people? What is your CacheStore configuration to avoid this?
>
> On Tue, May 2, 2017 at 11:50 AM, Jessie Lin 
> wrote:
>
>> Hello Matt, thank you for posting. I've noticed similar behavior.
>>
>> Would be curious to see the response from the engineering team.
>>
>> Best,
>> Jessie
>>
>> On Tue, May 2, 2017 at 1:03 AM, Matt  wrote:
>>
>>> Hi all,
>>>
>>> I have two questions for you!
>>>
>>> *QUESTION 1*
>>>
>>> I'm following the example in [1] (a mix between "jdbc transactional" and
>>> "jdbc bulk operations") and I've enabled write behind, however after the
>>> first 10k-20k insertions the performance drops *dramatically*.
>>>
>>> Based on prints I've added to the CacheStore, I've noticed what Ignite
>>> is doing is this:
>>>
>>> - writeAll called with 512 elements (Ignites buffers elements, that's
>>> good)
>>> - openConnection with autocommit=true is called each time inside
>>> writeAll (since session is not stored in atomic mode)
>>> - writeAll is called with 512 elements a few dozen times, each time it
>>> opens a new JDBC connection as mentioned above
>>> - ...
>>> - writeAll called with ONE element (for some reason Ignite stops
>>> buffering elements)
>>> - writeAll is called with ONE element from here on, each time it opens a
>>> new JDBC connection as mentioned above
>>> - ...
>>>
>>> Things to note:
>>>
>>> - All config values are the defaults ones except for write through and
>>> write behind which are both enabled.
>>> - I'm running this as a server node (only one node on the cluster, the
>>> application itself).
>>> - I see the problem even with a big heap (ie, Ignite is not nearly out
>>> of memory).
>>> - I'm using PostgreSQL for this test (it's fine ingesting around 40k
>>> rows per second on this computer, so that shouldn't be a problem)
>>>
>>> What is causing Ignite to stop buffering elements after calling
>>> writeAll() a few dozen times?
>>>
>>> *QUESTION 2*
>>>
>>> I've read on the docs that using ATOMIC mode (default mode) is better
>>> for performance, but I'm not getting why. If I'm not wrong using
>>> TRANSACTIONAL mode would cause the CacheStore to reuse connections (not
>>> call openConnection(autocommit=true) on each writeAll()).
>>>
>>> Shouldn't it be better to use transactional mode?
>>>
>>> Regards,
>>> Matt
>>>
>>> [1] https://apacheignite.readme.io/docs/persistent-store#sec
>>> tion-cachestore-example
>>>
>>
>>
>

Re: CacheStore's Performance Drops Dramatically - Why?

[Jessie Lin]

Hello Matt, thank you for posting. I've noticed similar behavior.

Would be curious to see the response from the engineering team.

Best,
Jessie

On Tue, May 2, 2017 at 1:03 AM, Matt  wrote:

> Hi all,
>
> I have two questions for you!
>
> *QUESTION 1*
>
> I'm following the example in [1] (a mix between "jdbc transactional" and
> "jdbc bulk operations") and I've enabled write behind, however after the
> first 10k-20k insertions the performance drops *dramatically*.
>
> Based on prints I've added to the CacheStore, I've noticed what Ignite is
> doing is this:
>
> - writeAll called with 512 elements (Ignites buffers elements, that's good)
> - openConnection with autocommit=true is called each time inside writeAll
> (since session is not stored in atomic mode)
> - writeAll is called with 512 elements a few dozen times, each time it
> opens a new JDBC connection as mentioned above
> - ...
> - writeAll called with ONE element (for some reason Ignite stops buffering
> elements)
> - writeAll is called with ONE element from here on, each time it opens a
> new JDBC connection as mentioned above
> - ...
>
> Things to note:
>
> - All config values are the defaults ones except for write through and
> write behind which are both enabled.
> - I'm running this as a server node (only one node on the cluster, the
> application itself).
> - I see the problem even with a big heap (ie, Ignite is not nearly out of
> memory).
> - I'm using PostgreSQL for this test (it's fine ingesting around 40k rows
> per second on this computer, so that shouldn't be a problem)
>
> What is causing Ignite to stop buffering elements after calling writeAll()
> a few dozen times?
>
> *QUESTION 2*
>
> I've read on the docs that using ATOMIC mode (default mode) is better for
> performance, but I'm not getting why. If I'm not wrong using TRANSACTIONAL
> mode would cause the CacheStore to reuse connections (not call
> openConnection(autocommit=true) on each writeAll()).
>
> Shouldn't it be better to use transactional mode?
>
> Regards,
> Matt
>
> [1] https://apacheignite.readme.io/docs/persistent-
> store#section-cachestore-example
>