[jira] [Updated] (IGNITE-2206) Make the file SecondaryFileSystemProvider pluggable
[ https://issues.apache.org/jira/browse/IGNITE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Ozerov updated IGNITE-2206: Attachment: (was: 2206.patch) > Make the file SecondaryFileSystemProvider pluggable > --- > > Key: IGNITE-2206 > URL: https://issues.apache.org/jira/browse/IGNITE-2206 > Project: Ignite > Issue Type: Sub-task > Components: hadoop, IGFS >Reporter: Ivan Veselovsky >Assignee: Vladimir Ozerov > Fix For: 1.6 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (IGNITE-2206) Make the file SecondaryFileSystemProvider pluggable
[ https://issues.apache.org/jira/browse/IGNITE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Ozerov resolved IGNITE-2206. - Resolution: Fixed > Make the file SecondaryFileSystemProvider pluggable > --- > > Key: IGNITE-2206 > URL: https://issues.apache.org/jira/browse/IGNITE-2206 > Project: Ignite > Issue Type: Sub-task > Components: hadoop, IGFS >Reporter: Ivan Veselovsky >Assignee: Vladimir Ozerov > Fix For: 1.6 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (IGNITE-2333) Hotspot in GridDhtPartitionTopologyImpl readLock() and readUnlock() methods
Vladimir Ozerov created IGNITE-2333: --- Summary: Hotspot in GridDhtPartitionTopologyImpl readLock() and readUnlock() methods Key: IGNITE-2333 URL: https://issues.apache.org/jira/browse/IGNITE-2333 Project: Ignite Issue Type: Sub-task Components: cache, general Affects Versions: ignite-1.4 Reporter: Vladimir Ozerov Fix For: 1.6 When running simple PUT benchmark in ATOMIC cache, read lock-unlock consumes up to 10% of time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (IGNITE-1371) Key-Value store (like Cassandra) as CacheStore
[ https://issues.apache.org/jira/browse/IGNITE-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082632#comment-15082632 ] Igor Rudyak edited comment on IGNITE-1371 at 1/5/16 8:25 AM: - Alexey, Items 1-3 are done and merged into ignite-1371 branch. Also merged changes from your latest review ignite-1371-review4. Will finish with the rest of the items in a couple of days. According items 6-7: 1) Persistence settings hierarchy was designed to allow persists Ignite Cache key and value using different settings. For example key could be persisted as BLOB, which will be mapped to appropriate BLOB field in Cassandra table, but value could be persisted as POJO, which will be mapped to a set of appropriate fields (int, long, string and etc.) in Cassandra table. Thus, depending on the use-case there could be different variations of how to persist key and value to provide better performance or usability (for reporting tools). 2) PersistenceStrategy specifies how you want to persist you Ignite Cache key and value into Cassandra database: - PRIMITIVE - could be used only for primitive Ignite Cache keys/values having simple java type (which could be directly mapped to appropriate Cassandra database type: int, long string, time and etc.). Thus Ignite Cache key/value will be save to appropriate Cassandra table filed having specific type compatible with java type (int, long, string and etc.) - BLOB - Ignite Cache keys/values will be saved into appropriate Cassandra table field having BLOB type. Conversion of key/value into BLOB is controlled by serializer which is specified. By default JavaSerializer will be used, which just using standard Java serialization to convert object to BLOB. - POJO - Stores each field of Ignite Cache key/value as a separate column in Cassandra table (having type which could be directly mapped to java type). Provides ability to utilize Cassandra secondary indexes for object fields. Could be used only for POJO objects following Java Beans convention. Actually I also developed rather detailed documentation for the module (with lot's of sampes). You can find it in the wiki: https://github.com/irudyak/ignite/wiki The same wiki URL was also added as a link to ignite-1371 feature and you can see it at the top of it. was (Author: irudyak): Alexey, Items 1-3 are done and merged into ignite-1371 branch. Also merged changed from your latest review ignite-1371-review4. Will finish with the rest of the items in a couple of days. According items 6-7: 1) Persistence settings hierarchy was designed to allow persists Ignite Cache key and value using different settings. For example key could be persisted as BLOB, which will be mapped to appropriate BLOB field in Cassandra table, but value could be persisted as POJO, which will be mapped to a set of appropriate fields (int, long, string and etc.) in Cassandra table. Thus, depending on the use-case there could be different variations of how to persist key and value to provide better performance or usability (for reporting tools). 2) PersistenceStrategy specifies how you want to persist you Ignite Cache key and value into Cassandra database: - PRIMITIVE - could be used only for primitive Ignite Cache keys/values having simple java type (which could be directly mapped to appropriate Cassandra database type: int, long string, time and etc.). Thus Ignite Cache key/value will be save to appropriate Cassandra table filed having specific type compatible with java type (int, long, string and etc.) - BLOB - Ignite Cache keys/values will be saved into appropriate Cassandra table field having BLOB type. Conversion of key/value into BLOB is controlled by serializer which is specified. By default JavaSerializer will be used, which just using standard Java serialization to convert object to BLOB. - POJO - Stores each field of Ignite Cache key/value as a separate column in Cassandra table (having type which could be directly mapped to java type). Provides ability to utilize Cassandra secondary indexes for object fields. Could be used only for POJO objects following Java Beans convention. Actually I also developed rather detailed documentation for the module (with lot's of sampes). You can find it in the wiki: https://github.com/irudyak/ignite/wiki The same wiki URL was also added as a link to ignite-1371 feature and you can see it at the top of it. > Key-Value store (like Cassandra) as CacheStore > -- > > Key: IGNITE-1371 > URL: https://issues.apache.org/jira/browse/IGNITE-1371 > Project: Ignite > Issue Type: New Feature > Components: cache >Affects Versions: ignite-1.4 >Reporter: Alexandre Boudnik >Assignee: Igor Rudyak > Attachments: master_02b59e4_ignite-1371.patch > >
[jira] [Commented] (IGNITE-2333) Hotspot in GridDhtPartitionTopologyImpl readLock() and readUnlock() methods
[ https://issues.apache.org/jira/browse/IGNITE-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082630#comment-15082630 ] Vladimir Ozerov commented on IGNITE-2333: - Attempted to replace RW lock with normal lock. Results are silly - PUT throughput is decreased in several times. Probably we need to employ striped locking here. > Hotspot in GridDhtPartitionTopologyImpl readLock() and readUnlock() methods > --- > > Key: IGNITE-2333 > URL: https://issues.apache.org/jira/browse/IGNITE-2333 > Project: Ignite > Issue Type: Sub-task > Components: cache, general >Affects Versions: ignite-1.4 >Reporter: Vladimir Ozerov > Fix For: 1.6 > > > When running simple PUT benchmark in ATOMIC cache, read lock-unlock consumes > up to 10% of time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-2333) Hotspot in GridDhtPartitionTopologyImpl readLock() and readUnlock() methods
[
https://issues.apache.org/jira/browse/IGNITE-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082693#comment-15082693
]
Vladimir Ozerov commented on IGNITE-2333:
-
Quick and dirty prototype. Shown ~10-15% increase in throughput for single-node
PUT benchmark.
{code}
public class StripedRWL implements ReadWriteLock {
private final PaddedRWL[] locks;
public StripedRWL() {
locks = new PaddedRWL[16];
for (int i = 0; i < locks.length; i++)
locks[i] = new PaddedRWL();
}
private static final ThreadLocal IDX = new ThreadLocal() {
@Override protected Integer initialValue() {
return ThreadLocalRandom.current().nextInt(10) % 16;
}
};
@NotNull @Override public Lock readLock() {
return locks[IDX.get()].readLock();
}
@NotNull @Override public Lock writeLock() {
return locks[IDX.get()].writeLock();
}
private static class PaddedRWL extends ReentrantReadWriteLock {
long p0, p1, p2, p3, p4, p5, p6, p7;
}
}
{code}
> Hotspot in GridDhtPartitionTopologyImpl readLock() and readUnlock() methods
> ---
>
> Key: IGNITE-2333
> URL: https://issues.apache.org/jira/browse/IGNITE-2333
> Project: Ignite
> Issue Type: Sub-task
> Components: cache, general
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
> Fix For: 1.6
>
>
> When running simple PUT benchmark in ATOMIC cache, read lock-unlock consumes
> up to 10% of time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (IGNITE-2335) Review ReentrantReadWriteLock usages.
Vladimir Ozerov created IGNITE-2335: --- Summary: Review ReentrantReadWriteLock usages. Key: IGNITE-2335 URL: https://issues.apache.org/jira/browse/IGNITE-2335 Project: Ignite Issue Type: Sub-task Components: general Affects Versions: ignite-1.4 Reporter: Vladimir Ozerov Fix For: 1.6 We have about 40-50 ReentrantReadWriteLock lock usages all over the project. This class is known to perform poorly under high readers contention. One typical example - IGNITE-2333. We need to do the following: 1) If lock has "busy" semantics (read.tryLock + read.unlock + write.lock), then change it to normal busy lock. 2) If lock appears to be on a hot path, it should be either replaced with correctly stripped/padded counterpart, or with simple ReentrantLock if protected operations are very small (e.g. single put into map). 3) Do not change anything otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (IGNITE-2334) Consider getting rid of XXX8 data structures.
Vladimir Ozerov created IGNITE-2334: --- Summary: Consider getting rid of XXX8 data structures. Key: IGNITE-2334 URL: https://issues.apache.org/jira/browse/IGNITE-2334 Project: Ignite Issue Type: Task Components: cache Affects Versions: ignite-1.4 Reporter: Vladimir Ozerov Fix For: 1.6 Current state of Java 8 data structures differs from our "jsr166" data structures. One noticeable change is how size is handled inside ConcurrentHashMap. Instead of using LongAdder, it has some inlined version of adder which is more suitable for map operations. But our outdated implementation ConcurrentHashMap8 still uses LongAdder8 and it is seen as a hotspot in profiler. We should evaluate whether it still makes sense to keep our versions of concurrent structures. 1) Quickly replace all *8* data structures with their normal counterparts. 2) Run benchmarks with Java7 and Java8. 3) Decide whether change is needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-949) Add Python API for Ignite RDD
[ https://issues.apache.org/jira/browse/IGNITE-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083360#comment-15083360 ] Dmitriy Setrakyan commented on IGNITE-949: -- Some useful points on the dev list: http://apache-ignite-developers.2346864.n4.nabble.com/Need-help-with-JIRA-ticket-td6399.html > Add Python API for Ignite RDD > - > > Key: IGNITE-949 > URL: https://issues.apache.org/jira/browse/IGNITE-949 > Project: Ignite > Issue Type: Task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Chandresh Pancholi > > Should be close to the Java version: > https://apacheignite.readme.io/docs/ignitecontext--igniterdd -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (IGNITE-429) Implement IgniteStormStreamer to stream data from Apache Storm
[
https://issues.apache.org/jira/browse/IGNITE-429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083076#comment-15083076
]
Roman Shtykh commented on IGNITE-429:
-
Fixed the above-mention limitation. Storm parallelization is usable now.
> Implement IgniteStormStreamer to stream data from Apache Storm
> --
>
> Key: IGNITE-429
> URL: https://issues.apache.org/jira/browse/IGNITE-429
> Project: Ignite
> Issue Type: Sub-task
> Components: streaming
>Reporter: Dmitriy Setrakyan
>Assignee: Roman Shtykh
> Fix For: 1.6
>
>
> We have {{IgniteDataStreamer}} which is used to load data into Ignite under
> high load. It was previously named {{IgniteDataLoader}}, see ticket
> IGNITE-394.
> See [Apache Storm|https://storm.apache.org/] for more information.
> We should create {{IgniteStormStreamer}} which will consume tuples from Storm
> and stream them into Ignite caches.
> More details to follow, but to the least we should be able to:
> - Convert/Intercept Storm tuples to Ignite data using an optional pluggable
> converter.
> - Specify the cache name for the Ignite cache to load data into.
> - Specify other flags available on {{IgniteDataStreamer}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (IGNITE-2206) Make the file SecondaryFileSystemProvider pluggable
[ https://issues.apache.org/jira/browse/IGNITE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Ozerov updated IGNITE-2206: Labels: important (was: ) > Make the file SecondaryFileSystemProvider pluggable > --- > > Key: IGNITE-2206 > URL: https://issues.apache.org/jira/browse/IGNITE-2206 > Project: Ignite > Issue Type: Sub-task > Components: hadoop, IGFS >Reporter: Ivan Veselovsky >Assignee: Vladimir Ozerov > Labels: important > Fix For: 1.6 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (IGNITE-1575) NPE when cache is started concurrently with the node stop
[
https://issues.apache.org/jira/browse/IGNITE-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Kulichenko updated IGNITE-1575:
Labels: (was: usability)
> NPE when cache is started concurrently with the node stop
> -
>
> Key: IGNITE-1575
> URL: https://issues.apache.org/jira/browse/IGNITE-1575
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Reporter: Valentin Kulichenko
>Priority: Minor
> Fix For: 1.6
>
>
> It's not causing any harm, but it's possible to get the NPE below during the
> node stop.
> {noformat}
> 57724 [main] ERROR IgniteKernal%t1-1 - Got exception while starting (will
> rollback startup routine).
> java.lang.NullPointerException
> at
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onKernalStart0(CacheContinuousQueryManager.java:91)
> at
> org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.onKernalStart(GridCacheManagerAdapter.java:97)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:1058)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:833)
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:829)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1549)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1416)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:916)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:477)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:458)
> at org.apache.ignite.Ignition.start(Ignition.java:321)
> .. application stack frames ...
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (IGNITE-1575) NPE when cache is started concurrently with the node stop
[
https://issues.apache.org/jira/browse/IGNITE-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Kulichenko updated IGNITE-1575:
Flags: Important
Priority: Major (was: Minor)
> NPE when cache is started concurrently with the node stop
> -
>
> Key: IGNITE-1575
> URL: https://issues.apache.org/jira/browse/IGNITE-1575
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Reporter: Valentin Kulichenko
> Fix For: 1.6
>
>
> It's not causing any harm, but it's possible to get the NPE below during the
> node stop.
> {noformat}
> 57724 [main] ERROR IgniteKernal%t1-1 - Got exception while starting (will
> rollback startup routine).
> java.lang.NullPointerException
> at
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onKernalStart0(CacheContinuousQueryManager.java:91)
> at
> org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.onKernalStart(GridCacheManagerAdapter.java:97)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:1058)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:833)
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:829)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1549)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1416)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:916)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:477)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:458)
> at org.apache.ignite.Ignition.start(Ignition.java:321)
> .. application stack frames ...
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
