[
https://issues.apache.org/jira/browse/SAMZA-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14051691#comment-14051691
]
Chris Riccomini commented on SAMZA-256:
---------------------------------------
bq. This would suggest that it would be better to put the in-memory KV store in
samza-core, since it doesn't have any library dependencies (apart from perhaps
Guava, if we choose to use that).
# If we use Guava, I don't want that dependency floating around in core. Google
is notoriously evil with their guava/collections garbage.
# If we opt to put the in-memory implementation in core, we'd have to move the
new abstract kv storage engine that [~cpsoman] is working on, as well. I looked
at the imports, and all the dependencies seem to be in API or core, so this
should be OK:
{code}
import java.io.File
import org.apache.samza.config.Config
import org.apache.samza.container.SamzaContainerContext
import org.apache.samza.serializers._
import org.apache.samza.SamzaException
import org.apache.samza.metrics.MetricsRegistry
import org.apache.samza.task.MessageCollector
import org.apache.samza.system.SystemStreamPartition
import org.apache.samza.storage.StorageEngineFactory
import org.apache.samza.storage.StorageEngine
{code}
If there's nothing that I've missed, then perhaps this is a good idea. As
Martin says, we could eliminate samza-kv all together, and just split it up
between samza-kv-* and samza-core.
bq. By the same logic, the LevelDB storage engine should be in a samza-leveldb
module, the RocksDB storage engine in a samza-rocksdb module, etc. Perhaps we
can remove the samza-kv module in 0.8.0 in favour of modules that make the
dependencies more explicit.
+1 If we can do this without adding any new dependencies to samza-core, I'm all
for this.
> Provide in-memory data store implementation
> -------------------------------------------
>
> Key: SAMZA-256
> URL: https://issues.apache.org/jira/browse/SAMZA-256
> Project: Samza
> Issue Type: Improvement
> Components: kv
> Affects Versions: 0.6.0
> Reporter: Jakob Homan
> Assignee: Chinmay Soman
> Fix For: 0.8.0
>
>
> The sole current kv store, LevelDbKeyValueStore, works well when the amount
> of data to be stored is prohibitively large to keep it all in memory.
> However, in cases where the state is small enough to comfortably fit in
> whatever memory is available, it would be better to provide an in-memory
> implementation. This can be backed by either a native Java class, or perhaps
> a Guava class, if that is found to scale better (or, of course, the backing
> implementation could be configurable).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
