[jira] [Commented] (SAMZA-969) allow empty serde for SystemStream for SystemConsumer
[ https://issues.apache.org/jira/browse/SAMZA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337232#comment-15337232 ] Yi Pan (Data Infrastructure) commented on SAMZA-969: RB is here: https://reviews.apache.org/r/48862/ > allow empty serde for SystemStream for SystemConsumer > - > > Key: SAMZA-969 > URL: https://issues.apache.org/jira/browse/SAMZA-969 > Project: Samza > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Fix For: 0.10.1 > > Attachments: patch > > > If serde is configured as "" (empty string), we shouldn't create a serde and > that will just pass the object "as is" through the SerdeManager. > SAMZA-959 covered serde specified for systems. > This one covers serde specified for SystemStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-969) allow empty serde for SystemStream for SystemConsumer
[ https://issues.apache.org/jira/browse/SAMZA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337214#comment-15337214 ] Yi Pan (Data Infrastructure) commented on SAMZA-969: +1. Merged and submitted. Thanks! > allow empty serde for SystemStream for SystemConsumer > - > > Key: SAMZA-969 > URL: https://issues.apache.org/jira/browse/SAMZA-969 > Project: Samza > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Fix For: 0.10.1 > > Attachments: patch > > > If serde is configured as "" (empty string), we shouldn't create a serde and > that will just pass the object "as is" through the SerdeManager. > SAMZA-959 covered serde specified for systems. > This one covers serde specified for SystemStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-727: --- Attachment: SAMZA-727-9.patch Fixed some license and checkstyle issues from SMAZA-727-8.patch. Ready for merge. > Support for Kerberos > > > Key: SAMZA-727 > URL: https://issues.apache.org/jira/browse/SAMZA-727 > Project: Samza > Issue Type: New Feature > Components: yarn >Affects Versions: 0.9.0 > Environment: YARN with Kerberos >Reporter: Qi FU >Assignee: Chen Song > Attachments: SAMZA-727-9.patch, SAMZA-727.5.patch, SAMZA-727.7.patch, > SAMZA-727.patch, SMAZA-727.2.patch, SMAZA-727.6.patch, SMAZA-727.8.patch > > > Samza doesn't support Kerberos, which is very common for YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-855) Upgrade Samza's Kafka client version to 0.9.0.0
[ https://issues.apache.org/jira/browse/SAMZA-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-855: --- Assignee: Robert Crim > Upgrade Samza's Kafka client version to 0.9.0.0 > --- > > Key: SAMZA-855 > URL: https://issues.apache.org/jira/browse/SAMZA-855 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) >Assignee: Robert Crim > > As Kafka 0.9.0.0 is out, we should upgrade Samza's client library to 0.9.0.0 > to include the new consumer library and the bug fixes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-957) Avoid unnecessary KV Store flushes (part 3)
[
https://issues.apache.org/jira/browse/SAMZA-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319620#comment-15319620
]
Yi Pan (Data Infrastructure) commented on SAMZA-957:
Merged and submitted. Thanks!
> Avoid unnecessary KV Store flushes (part 3)
> ---
>
> Key: SAMZA-957
> URL: https://issues.apache.org/jira/browse/SAMZA-957
> Project: Samza
> Issue Type: Bug
>Reporter: Jake Maes
>Assignee: Jake Maes
> Fix For: 0.10.1
>
> Attachments: SAMZA-957_1.patch
>
>
> We had an issue where RocksDB performance severely degraded for 23 hours and
> then resolved itself. To troubleshoot the issue I gathered some samples of
> the compaction stats from the RocksDB log and engaged with the RocksDB team
> via an existing, related issue:
> https://github.com/facebook/rocksdb/issues/696#issuecomment-222549220
> They pointed out that the job was flushing excessively:
> {quote}
> If you overload RocksDB with work (i.e. do bunch of writes really fast, or in
> your case, bunch of small flushes), it will begin stalling writes while the
> compactions (deferred work) completes. An interesting thing with RocksDB and
> LSM architecture is that the more behind you are on compactions, the more
> expensive the compactions are (due to increased write amplifications and
> single-threadness of L0->L1 compaction). So our write stalls have to be tuned
> exactly right for RocksDB to behave well with extremely high write rate.
> {quote}
> Looking through our commit history I see that SAMZA-812 and SAMZA-873 have
> both intended to address this issue, by reducing the amount of flushes in
> CachedStore.
> To be fair, the job in question did not have the SAMZA-873 patch, but I see
> even more room for improvement. Namely, CachedStore should *never* flush the
> underlying store unless its flush() was called. It can purge its dirty items
> to trade off performance for correctness, but flushing is excessive. So, this
> patch will remove the flushes from the all() and range() methods, simplify
> the LRU logic, and add a good unit test to verify and explain the proper LRU
> behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SAMZA-855) Upgrade Samza's Kafka client version to 0.9.0.0
[ https://issues.apache.org/jira/browse/SAMZA-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318884#comment-15318884 ] Yi Pan (Data Infrastructure) commented on SAMZA-855: [~rjc...@gmail.com], glad that you want to take it up! Really appreciate it! There was an email discussion on the potential compatibility issues. Some of Samza community users are still using Kafka 0.8.2 broker versions, which may be an issue if Samza's Kafka client version is 0.9 and above. [~cpsoman], could you comment on when Uber will be move to Kafka 0.9 and above on the broker side? One potential way to maintain backward compatibility is to: # upgrade the default KafkaSystemFactory and corresponding consumer/producers to Kafka 0.9 # keep the current KafkaSystemFactory as OldKakfaSystemFactory and rename the corresponding consumer/producers to OldKafkaSystemConsumer and OldKafkaSystemProducer for users who are not ready to move to newer version of Kafka on the broker side Thoughts? > Upgrade Samza's Kafka client version to 0.9.0.0 > --- > > Key: SAMZA-855 > URL: https://issues.apache.org/jira/browse/SAMZA-855 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) > > As Kafka 0.9.0.0 is out, we should upgrade Samza's client library to 0.9.0.0 > to include the new consumer library and the bug fixes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-727: --- Attachment: SAMZA-727.7.patch [~capricornius], I have updated the patch against latest master and have made some minor modifications as I commented on the RB. Could you please take a look and verify the change? Thanks a lot! > Support for Kerberos > > > Key: SAMZA-727 > URL: https://issues.apache.org/jira/browse/SAMZA-727 > Project: Samza > Issue Type: New Feature > Components: yarn >Affects Versions: 0.9.0 > Environment: YARN with Kerberos >Reporter: Qi FU >Assignee: Chen Song > Attachments: SAMZA-727.5.patch, SAMZA-727.7.patch, SAMZA-727.patch, > SMAZA-727.2.patch, SMAZA-727.6.patch > > > Samza doesn't support Kerberos, which is very common for YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-680) Invert the JobCoordinator and AM logic
[
https://issues.apache.org/jira/browse/SAMZA-680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313147#comment-15313147
]
Yi Pan (Data Infrastructure) commented on SAMZA-680:
Merged and submitted. Thanks!
> Invert the JobCoordinator and AM logic
> --
>
> Key: SAMZA-680
> URL: https://issues.apache.org/jira/browse/SAMZA-680
> Project: Samza
> Issue Type: Sub-task
>Reporter: Naveen Somasundaram
>Assignee: Jagadish
> Attachments: rb47687 (1).patch
>
>
> Currently, the YARN AM pretty much dictates how the JobCoordinator works.
> This creates lot of inflexibility on how we can control failures or even
> integrate with new system (Mesos).
> For e.g.,
> https://issues.apache.org/jira/browse/SAMZA-465?focusedCommentId=14522043=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14522043
> It would nice to invert the logic to JobCoordinator, and JobCoordinator has a
> global view of container failures, config changes etc. This simplifies lot of
> implementation specifics (for e.g., dynamic scaling becomes easier).
> A another nice to have, would be make this logic pluggable.
> {noformat}
> e.g.,
> job.clustermanager = org.apache.samza.YarnClusterManager
> job.clustermanager = org.apache.samza.MesosClusterManager
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SAMZA-955) Upgrade RocksDB JNI to 4.5.1
Yi Pan (Data Infrastructure) created SAMZA-955: -- Summary: Upgrade RocksDB JNI to 4.5.1 Key: SAMZA-955 URL: https://issues.apache.org/jira/browse/SAMZA-955 Project: Samza Issue Type: Bug Reporter: Yi Pan (Data Infrastructure) It seems that there are benefits to move to rocksdbjni-4.5.1, which solves some native library linking issues (e.g. SAMZA-870) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-870) RocksDBException: Invalid argument: Compression type Snappy is not linked with the binary
[ https://issues.apache.org/jira/browse/SAMZA-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304941#comment-15304941 ] Yi Pan (Data Infrastructure) commented on SAMZA-870: [~staslos], thanks for confirming! I got a response from RocksDB community via [issue-1127|https://github.com/facebook/rocksdb/issues/1127] that the later versions after rocksdbjni-3.13.1 resolves this problem. I have opened SAMZA-955 to track the effort to upgrade to rocksdbjni-4.5.1. > RocksDBException: Invalid argument: Compression type Snappy is not linked > with the binary > - > > Key: SAMZA-870 > URL: https://issues.apache.org/jira/browse/SAMZA-870 > Project: Samza > Issue Type: Bug > Components: kv-store >Affects Versions: 0.10.1 > Environment: RedHat Enterprise Linux 6.6, Java 8 (Oracle 1.8.0_40) >Reporter: Edi Bice >Assignee: Navina Ramesh > > I'm running latest from master with a small patch to allow for additional > RocksDB configuration. All my jobs (which involve KV stores) fail with the > following: > org.apache.samza.SamzaException: Error opening RocksDB store > xxx-store/Partition_6, received the following exception from RocksDB > org.rocksdb.RocksDBException: Invalid argument: Compression type Snappy is > not linked with the binary. > at > org.apache.samza.storage.kv.RocksDbKeyValueStore$.openDB(RocksDbKeyValueStore.scala:81) > at > org.apache.samza.storage.kv.RocksDbKeyValueStore.db$lzycompute(RocksDbKeyValueStore.scala:103) > at > org.apache.samza.storage.kv.RocksDbKeyValueStore.db(RocksDbKeyValueStore.scala:103) > at > org.apache.samza.storage.kv.RocksDbKeyValueStore.flush(RocksDbKeyValueStore.scala:195) > at > org.apache.samza.storage.kv.LoggedStore.flush(LoggedStore.scala:107) > at > org.apache.samza.storage.kv.SerializedKeyValueStore.flush(SerializedKeyValueStore.scala:117) > at > org.apache.samza.storage.kv.CachedStore.flush(CachedStore.scala:194) > at > org.apache.samza.storage.kv.NullSafeKeyValueStore.flush(NullSafeKeyValueStore.scala:82) > at > org.apache.samza.storage.kv.KeyValueStorageEngine.flush(KeyValueStorageEngine.scala:123) > at > org.apache.samza.storage.kv.KeyValueStorageEngine.close(KeyValueStorageEngine.scala:135) > at > org.apache.samza.storage.kv.KeyValueStorageEngine.stop(KeyValueStorageEngine.scala:129) > at > org.apache.samza.storage.TaskStorageManager$$anonfun$stopStores$2.apply(TaskStorageManager.scala:193) > at > org.apache.samza.storage.TaskStorageManager$$anonfun$stopStores$2.apply(TaskStorageManager.scala:193) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at > scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206) > at > org.apache.samza.storage.TaskStorageManager.stopStores(TaskStorageManager.scala:193) > at > org.apache.samza.storage.TaskStorageManager.stop(TaskStorageManager.scala:197) > at > org.apache.samza.container.TaskInstance.shutdownStores(TaskInstance.scala:203) > at > org.apache.samza.container.SamzaContainer$$anonfun$shutdownStores$2.apply(SamzaContainer.scala:689) > at > org.apache.samza.container.SamzaContainer$$anonfun$shutdownStores$2.apply(SamzaContainer.scala:689) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at > scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206) > at > org.apache.samza.container.SamzaContainer.shutdownStores(SamzaContainer.scala:689) > at > org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:568) > at > org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93) > at > org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67) > at > org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302985#comment-15302985 ] Yi Pan (Data Infrastructure) commented on SAMZA-727: [~capricornius], Thanks a lot! I have some comments on the latest RB for this one and SAMZA-928. Let me know if you need help in updating the RBs. Thanks! > Support for Kerberos > > > Key: SAMZA-727 > URL: https://issues.apache.org/jira/browse/SAMZA-727 > Project: Samza > Issue Type: New Feature > Components: yarn >Affects Versions: 0.9.0 > Environment: YARN with Kerberos >Reporter: Qi FU >Assignee: Chen Song > Attachments: SAMZA-727.5.patch, SAMZA-727.patch, SMAZA-727.2.patch, > SMAZA-727.6.patch > > > Samza doesn't support Kerberos, which is very common for YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-951) Improve event loop timing metrics
[ https://issues.apache.org/jira/browse/SAMZA-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299227#comment-15299227 ] Yi Pan (Data Infrastructure) commented on SAMZA-951: Merged and submitted. Thanks! > Improve event loop timing metrics > - > > Key: SAMZA-951 > URL: https://issues.apache.org/jira/browse/SAMZA-951 > Project: Samza > Issue Type: Improvement > Components: metrics >Affects Versions: 0.10.1 >Reporter: Jake Maes >Assignee: Jake Maes >Priority: Minor > Fix For: 0.10.1 > > Attachments: SAMZA-951_1.patch, SAMZA-951_2.patch, SAMZA-951_3.patch > > > SAMZA-401 added the event-loop-utilization metric to measure the % active > time in the event loop. However, the value for the event-loop-utilization (at > least since samza 10) is almost always ~= 1.0, even for over-provisioned > jobs, which is not very useful. > The way it is currently measured, the active time includes choose-ns which > includes time spent in poll() waiting for new messages. This waiting should > be considered idle time. > The goals of this ticket are to: > 1. Exclude the choose time from the active time to fix the > event-loop-utilization metric. > 2. Add deserialization-ns and poll-ns metrics to help users understand where > the time is being spent in the chooser. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-949) KafkaCheckpointManager should not try forever for non-recoverable checkpoint deserialization failures
[ https://issues.apache.org/jira/browse/SAMZA-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298927#comment-15298927 ] Yi Pan (Data Infrastructure) commented on SAMZA-949: [~TaoFeng], it is related but different issue. This JIRA is tracking a more general issue that deserialization or any non-recoverable failure in the checkpoint topic should not be re-tried, while SAMZA-807 is *only* targeted to deserialization errors caused by grouper class validation. > KafkaCheckpointManager should not try forever for non-recoverable checkpoint > deserialization failures > - > > Key: SAMZA-949 > URL: https://issues.apache.org/jira/browse/SAMZA-949 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) >Assignee: Branislav Cogic > > Currently, KafkaCheckpointManager retries forever, if there is a > deserialization error from checkpoint topic. One example is that the > deserialization failed due to validation of the "same" grouper factory is > used in between two job runs. We should differentiate the non-recoverable > case and error out here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-870) RocksDBException: Invalid argument: Compression type Snappy is not linked with the binary
[ https://issues.apache.org/jira/browse/SAMZA-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292156#comment-15292156 ] Yi Pan (Data Infrastructure) commented on SAMZA-870: Jose and [~edi_b...@yahoo.com], I have opened an issue in rocksDB community to track this issue. We did not see this issue after we got an workaround for our unit test working for [issue-989|https://github.com/facebook/rocksdb/issues/989]. The issue I opened is [issue-1127|https://github.com/facebook/rocksdb/issues/1127]. > RocksDBException: Invalid argument: Compression type Snappy is not linked > with the binary > - > > Key: SAMZA-870 > URL: https://issues.apache.org/jira/browse/SAMZA-870 > Project: Samza > Issue Type: Bug > Components: kv-store >Affects Versions: 0.10.1 > Environment: RedHat Enterprise Linux 6.6, Java 8 (Oracle 1.8.0_40) >Reporter: Edi Bice >Assignee: Navina Ramesh > > I'm running latest from master with a small patch to allow for additional > RocksDB configuration. All my jobs (which involve KV stores) fail with the > following: > org.apache.samza.SamzaException: Error opening RocksDB store > xxx-store/Partition_6, received the following exception from RocksDB > org.rocksdb.RocksDBException: Invalid argument: Compression type Snappy is > not linked with the binary. > at > org.apache.samza.storage.kv.RocksDbKeyValueStore$.openDB(RocksDbKeyValueStore.scala:81) > at > org.apache.samza.storage.kv.RocksDbKeyValueStore.db$lzycompute(RocksDbKeyValueStore.scala:103) > at > org.apache.samza.storage.kv.RocksDbKeyValueStore.db(RocksDbKeyValueStore.scala:103) > at > org.apache.samza.storage.kv.RocksDbKeyValueStore.flush(RocksDbKeyValueStore.scala:195) > at > org.apache.samza.storage.kv.LoggedStore.flush(LoggedStore.scala:107) > at > org.apache.samza.storage.kv.SerializedKeyValueStore.flush(SerializedKeyValueStore.scala:117) > at > org.apache.samza.storage.kv.CachedStore.flush(CachedStore.scala:194) > at > org.apache.samza.storage.kv.NullSafeKeyValueStore.flush(NullSafeKeyValueStore.scala:82) > at > org.apache.samza.storage.kv.KeyValueStorageEngine.flush(KeyValueStorageEngine.scala:123) > at > org.apache.samza.storage.kv.KeyValueStorageEngine.close(KeyValueStorageEngine.scala:135) > at > org.apache.samza.storage.kv.KeyValueStorageEngine.stop(KeyValueStorageEngine.scala:129) > at > org.apache.samza.storage.TaskStorageManager$$anonfun$stopStores$2.apply(TaskStorageManager.scala:193) > at > org.apache.samza.storage.TaskStorageManager$$anonfun$stopStores$2.apply(TaskStorageManager.scala:193) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at > scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206) > at > org.apache.samza.storage.TaskStorageManager.stopStores(TaskStorageManager.scala:193) > at > org.apache.samza.storage.TaskStorageManager.stop(TaskStorageManager.scala:197) > at > org.apache.samza.container.TaskInstance.shutdownStores(TaskInstance.scala:203) > at > org.apache.samza.container.SamzaContainer$$anonfun$shutdownStores$2.apply(SamzaContainer.scala:689) > at > org.apache.samza.container.SamzaContainer$$anonfun$shutdownStores$2.apply(SamzaContainer.scala:689) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at > scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206) > at > org.apache.samza.container.SamzaContainer.shutdownStores(SamzaContainer.scala:689) > at > org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:568) > at > org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93) > at > org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67) > at > org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-950) Add backward-compatible KafkaSystemProducer constructor for Java clients
[ https://issues.apache.org/jira/browse/SAMZA-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292115#comment-15292115 ] Yi Pan (Data Infrastructure) commented on SAMZA-950: [~jmakes], merged and submitted. Thanks! > Add backward-compatible KafkaSystemProducer constructor for Java clients > > > Key: SAMZA-950 > URL: https://issues.apache.org/jira/browse/SAMZA-950 > Project: Samza > Issue Type: Bug >Affects Versions: 0.10.1 >Reporter: Jake Maes >Assignee: Jake Maes > Fix For: 0.10.1 > > Attachments: SAMZA-950_1.patch, SAMZA-950_2.patch > > > SAMZA-911 made a change to the constructor of KafkaSystemProducer. > While that change is backward-compatible for Scala clients, it is not for > Java clients. > Although KafkaSystemProducer is not part of our public API, we have a team at > LI that uses it in Java and we want to avoid breaking their code on a minor > version bump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SAMZA-949) KafkaCheckpointManager should not try forever for non-recoverable checkpoint deserialization failures
Yi Pan (Data Infrastructure) created SAMZA-949: -- Summary: KafkaCheckpointManager should not try forever for non-recoverable checkpoint deserialization failures Key: SAMZA-949 URL: https://issues.apache.org/jira/browse/SAMZA-949 Project: Samza Issue Type: Bug Reporter: Yi Pan (Data Infrastructure) Currently, KafkaCheckpointManager retries forever, if there is a deserialization error from checkpoint topic. One example is that the deserialization failed due to validation of the "same" grouper factory is used in between two job runs. We should differentiate the non-recoverable case and error out here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-948) CoordinatorStreamSystemConsumer is not threadsafe.
[ https://issues.apache.org/jira/browse/SAMZA-948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282226#comment-15282226 ] Yi Pan (Data Infrastructure) commented on SAMZA-948: Merged and submitted. Thanks! > CoordinatorStreamSystemConsumer is not threadsafe. > -- > > Key: SAMZA-948 > URL: https://issues.apache.org/jira/browse/SAMZA-948 > Project: Samza > Issue Type: Bug >Affects Versions: 0.10.1 >Reporter: Jake Maes >Assignee: Jake Maes >Priority: Blocker > Fix For: 0.10.1 > > Attachments: SAMZA-948_1.patch, SAMZA-948_2.patch, SAMZA-948_3.patch, > SAMZA-948_4.patch, SAMZA-948_5.patch, SAMZA-948_6.patch > > > In testing the 10.1 release I found some ConcurrentModificationExceptions > resulting from the SAMZA-913 patch. > It appears that the AM UI, onContainerComplete callback, and probably other > codepaths can cause concurrent coordinator stream bootstraps, which was > always a problem, but now causes a ConcurrentModificationException because > bootstrap calls remove() on the bootstrap messages set. > Here are a couple stack traces illustrating the issue: > java.util.ConcurrentModificationException > at > java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711) > at > java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:734) > at > org.apache.samza.coordinator.stream.CoordinatorStreamSystemConsumer.getBootstrappedStream(CoordinatorStreamSystemConsumer.java:188) > at > org.apache.samza.coordinator.stream.AbstractCoordinatorStreamManager.getBootstrappedStream(AbstractCoordinatorStreamManager.java:85) > at > org.apache.samza.container.LocalityManager.readContainerLocality(LocalityManager.java:101) > at > org.apache.samza.job.model.JobModel.getContainerToHostValue(JobModel.java:96) > at > org.apache.samza.job.yarn.SamzaTaskManager.onContainerCompleted(SamzaTaskManager.java:210) > at > org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersCompleted$1$$anonfun$apply$5.apply(SamzaAppMaster.scala:143) > at > org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersCompleted$1$$anonfun$apply$5.apply(SamzaAppMaster.scala:143) > at scala.collection.immutable.List.foreach(List.scala:318) > at > org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersCompleted$1.apply(SamzaAppMaster.scala:143) > at > org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersCompleted$1.apply(SamzaAppMaster.scala:143) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > org.apache.samza.job.yarn.SamzaAppMaster$.onContainersCompleted(SamzaAppMaster.scala:143) > at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:287) > java.util.ConcurrentModificationException > at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711) > at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:734) > at > org.apache.samza.coordinator.stream.CoordinatorStreamSystemConsumer.getBootstrappedStream(CoordinatorStreamSystemConsumer.java:188) > at > org.apache.samza.coordinator.stream.AbstractCoordinatorStreamManager.getBootstrappedStream(AbstractCoordinatorStreamManager.java:85) > at > org.apache.samza.container.LocalityManager.readContainerLocality(LocalityManager.java:101) > at > org.apache.samza.job.model.JobModel.getContainerToHostValue(JobModel.java:96) > at > views.$_scalate_$index_scaml$$anonfun$$_scalate_$render$3.apply(index.scaml.scala:230) > at > views.$_scalate_$index_scaml$$anonfun$$_scalate_$render$3.apply(index.scaml.scala:183) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) > at views.$_scalate_$index_scaml.render(index.scaml.scala:331) > at > org.fusesource.scalate.DefaultRenderContext.capture(DefaultRenderContext.scala:92) > at > org.fusesource.scalate.layout.DefaultLayoutStrategy.layout(DefaultLayoutStrategy.scala:45) > at > org.fusesource.scalate.TemplateEngine$$anonfun$layout$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(TemplateEngine.scala:559) > at >
[jira] [Commented] (SAMZA-852) Better logging when system can not be created
[ https://issues.apache.org/jira/browse/SAMZA-852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281846#comment-15281846 ] Yi Pan (Data Infrastructure) commented on SAMZA-852: +1. Merged and submitted. Thanks! > Better logging when system can not be created > - > > Key: SAMZA-852 > URL: https://issues.apache.org/jira/browse/SAMZA-852 > Project: Samza > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Vadim Chekan >Assignee: Jake Maes >Priority: Minor > Fix For: 0.10.1 > > Attachments: > 0001-Inability-to-create-configured-producer-should-be-lo.patch, > SAMZA-852_2.patch > > > Inability to create configured producer should be logged as error. If not, it > leads to unneeded investigation "why does it say system X can not be found, > here it is!". > Also, do not drop the original exception, it makes troubleshooting more > difficult. > My pull request: https://github.com/apache/samza/pull/8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-947) TaskAssignmentManager registration exception when partition count changes.
[ https://issues.apache.org/jira/browse/SAMZA-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281601#comment-15281601 ] Yi Pan (Data Infrastructure) commented on SAMZA-947: +1. Merged and submitted. Thanks! > TaskAssignmentManager registration exception when partition count changes. > -- > > Key: SAMZA-947 > URL: https://issues.apache.org/jira/browse/SAMZA-947 > Project: Samza > Issue Type: Bug >Affects Versions: 0.10.1 >Reporter: Jake Maes >Assignee: Jake Maes >Priority: Minor > Fix For: 0.10.1 > > Attachments: SAMZA-947_1.patch, SAMZA-947_2.patch > > > The GroupByPartitionCount grouper deletes the persisted task mapping if the > partition count has changed because there may be fewer tasks and that would > cause the old mapping to be invalid. > To delete the mapping, the TaskAssignmentManager registers itself and writes > null for all the keys. Later when the recalculated mapping is saved, it tries > to reregister itself, which causes this exception: > Exception in thread "main" org.apache.samza.SamzaException: > SamzaTaskAssignmentManager is already registered with the queuing system > producer > at > org.apache.samza.system.kafka.KafkaSystemProducer.register(KafkaSystemProducer.scala:65) > at > org.apache.samza.coordinator.stream.CoordinatorStreamSystemProducer.register(CoordinatorStreamSystemProducer.java:72) > at > org.apache.samza.coordinator.stream.AbstractCoordinatorStreamManager.registerCoordinatorStreamProducer(AbstractCoordinatorStreamManager.java:100) > at > org.apache.samza.container.grouper.task.TaskAssignmentManager.register(TaskAssignmentManager.java:58) > at > org.apache.samza.container.grouper.task.GroupByContainerCount.saveTaskAssignments(GroupByContainerCount.java:179) > at > org.apache.samza.container.grouper.task.GroupByContainerCount.balance(GroupByContainerCount.java:93) > at > org.apache.samza.coordinator.JobCoordinator$.refreshJobModel(JobCoordinator.scala:255) > at > org.apache.samza.coordinator.JobCoordinator$.jobModelGenerator$1(JobCoordinator.scala:187) > at > org.apache.samza.coordinator.JobCoordinator$.initializeJobModel(JobCoordinator.scala:193) > at > org.apache.samza.coordinator.JobCoordinator$.getJobCoordinator(JobCoordinator.scala:120) > at > org.apache.samza.coordinator.JobCoordinator$.apply(JobCoordinator.scala:104) > at > org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:74) > at org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala) > In a YARN environment, the AM restarts and since the task mapping has now > been deleted, this 2nd attempt to save the mapping succeeds. > Since this issue only occurs when the partition count changes and is > recoverable, I'm marking it as low priority. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-929) Support AM restart in Kerberos secured YARN after the original token in launch context expires
[ https://issues.apache.org/jira/browse/SAMZA-929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261290#comment-15261290 ] Yi Pan (Data Infrastructure) commented on SAMZA-929: Hi, [~capricornius], if you have some test to verify it, it would be good to at least add the test setup here. Then we can close this one. Thanks! > Support AM restart in Kerberos secured YARN after the original token in > launch context expires > -- > > Key: SAMZA-929 > URL: https://issues.apache.org/jira/browse/SAMZA-929 > Project: Samza > Issue Type: Sub-task > Components: yarn >Reporter: Yi Pan (Data Infrastructure) >Assignee: Chen Song > > In Kerberos secured YARN cluster, an initial token is given in the launch > context to RM for a job. When that token expires, RM will not be able to > restart failed AM. As for now, we will have to re-submit the job to refresh > the token in the launch context. > This ticket is searching for a solution that can refresh the token in the > launch context for AM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-937) Correct URL to Coda's metrics library
[ https://issues.apache.org/jira/browse/SAMZA-937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-937: --- Assignee: Robin Tweedie > Correct URL to Coda's metrics library > - > > Key: SAMZA-937 > URL: https://issues.apache.org/jira/browse/SAMZA-937 > Project: Samza > Issue Type: Improvement > Components: docs >Reporter: Robin Tweedie >Assignee: Robin Tweedie >Priority: Trivial > Labels: documentation, newbie > Attachments: SAMZA-937.patch > > > It appears that the homepage for Coda Hale's metrics library is now at > http://metrics.dropwizard.io > Review: https://reviews.apache.org/r/46309/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15260976#comment-15260976 ] Yi Pan (Data Infrastructure) commented on SAMZA-727: Hi, [~capricornius], left some comment on [RB|https://reviews.apache.org/r/37026/]. Thanks! > Support for Kerberos > > > Key: SAMZA-727 > URL: https://issues.apache.org/jira/browse/SAMZA-727 > Project: Samza > Issue Type: New Feature > Components: yarn >Affects Versions: 0.9.0 > Environment: YARN with Kerberos >Reporter: Qi FU >Assignee: Chen Song > Attachments: SAMZA-727.patch, SMAZA-727.2.patch > > > Samza doesn't support Kerberos, which is very common for YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258759#comment-15258759 ] Yi Pan (Data Infrastructure) commented on SAMZA-727: Nevermind, I found the [RB|https://reviews.apache.org/r/37026]. Thanks! > Support for Kerberos > > > Key: SAMZA-727 > URL: https://issues.apache.org/jira/browse/SAMZA-727 > Project: Samza > Issue Type: New Feature > Components: yarn >Affects Versions: 0.9.0 > Environment: YARN with Kerberos >Reporter: Qi FU >Assignee: Chen Song > Attachments: SAMZA-727.patch, SMAZA-727.2.patch > > > Samza doesn't support Kerberos, which is very common for YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258613#comment-15258613 ] Yi Pan (Data Infrastructure) commented on SAMZA-727: [~capricornius], could you also create an RB for the patch? That would be easier for commenting/reviews. Thanks! > Support for Kerberos > > > Key: SAMZA-727 > URL: https://issues.apache.org/jira/browse/SAMZA-727 > Project: Samza > Issue Type: New Feature > Components: yarn >Affects Versions: 0.9.0 > Environment: YARN with Kerberos >Reporter: Qi FU >Assignee: Chen Song > Attachments: SAMZA-727.patch, SMAZA-727.2.patch > > > Samza doesn't support Kerberos, which is very common for YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-944) Broadcast stream is not added properly in the prioritized tiers in the DefaultChooser
[ https://issues.apache.org/jira/browse/SAMZA-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-944: --- Affects Version/s: 0.10.0 > Broadcast stream is not added properly in the prioritized tiers in the > DefaultChooser > - > > Key: SAMZA-944 > URL: https://issues.apache.org/jira/browse/SAMZA-944 > Project: Samza > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Yi Pan (Data Infrastructure) > Fix For: 0.10.1 > > > When bootstrap and / or prioritized input streams are configured, broadcast > streams are not added to the default prioritized tier in the DefaultChooser > properly. Hence, it is not consumed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-944) Broadcast stream is not added properly in the prioritized tiers in the DefaultChooser
[ https://issues.apache.org/jira/browse/SAMZA-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-944: --- Fix Version/s: 0.10.1 > Broadcast stream is not added properly in the prioritized tiers in the > DefaultChooser > - > > Key: SAMZA-944 > URL: https://issues.apache.org/jira/browse/SAMZA-944 > Project: Samza > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Yi Pan (Data Infrastructure) > Fix For: 0.10.1 > > > When bootstrap and / or prioritized input streams are configured, broadcast > streams are not added to the default prioritized tier in the DefaultChooser > properly. Hence, it is not consumed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SAMZA-942) Document the core set of metrics Samza generate
Yi Pan (Data Infrastructure) created SAMZA-942: -- Summary: Document the core set of metrics Samza generate Key: SAMZA-942 URL: https://issues.apache.org/jira/browse/SAMZA-942 Project: Samza Issue Type: Improvement Reporter: Yi Pan (Data Infrastructure) Currently, there is no explanation of the list of metrics that Samza framework generates. Hence, it is hard for the user to interpret the meaning and understand the runtime status of a job. We should document the core set of metrics that Samza tracks in the framework. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SAMZA-941) fix the bug containsValue method in MapConfig
Yi Pan (Data Infrastructure) created SAMZA-941: -- Summary: fix the bug containsValue method in MapConfig Key: SAMZA-941 URL: https://issues.apache.org/jira/browse/SAMZA-941 Project: Samza Issue Type: Bug Reporter: Yi Pan (Data Infrastructure) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-911) Abandon producer retry after a certain # of errors
[ https://issues.apache.org/jira/browse/SAMZA-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251291#comment-15251291 ] Yi Pan (Data Infrastructure) commented on SAMZA-911: [~jagadish1...@gmail.com], added comments to the RB. Thanks! > Abandon producer retry after a certain # of errors > -- > > Key: SAMZA-911 > URL: https://issues.apache.org/jira/browse/SAMZA-911 > Project: Samza > Issue Type: Bug >Reporter: Jagadish >Assignee: Jagadish > Fix For: 0.10.1 > > Attachments: SAMZA-911.0.patch > > > Currently, the KafkaSystemProducer's producer loop keeps retrying > indefinitely when there is an exception in the retryBackOff loop. If there > are repeated exceptions, then it makes sense to retry for awhile, and then > fail the container. > Long term, we should focus on getting rid off the retryBackOff loop, and > close the producer object in the callback during failure. This will guarantee > in-order delivery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-911) Abandon producer retry after a certain # of errors
[ https://issues.apache.org/jira/browse/SAMZA-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-911: --- Fix Version/s: 0.10.1 > Abandon producer retry after a certain # of errors > -- > > Key: SAMZA-911 > URL: https://issues.apache.org/jira/browse/SAMZA-911 > Project: Samza > Issue Type: Bug >Reporter: Jagadish >Assignee: Jagadish > Fix For: 0.10.1 > > Attachments: SAMZA-911.0.patch > > > Currently, the KafkaSystemProducer's producer loop keeps retrying > indefinitely when there is an exception in the retryBackOff loop. If there > are repeated exceptions, then it makes sense to retry for awhile, and then > fail the container. > Long term, we should focus on getting rid off the retryBackOff loop, and > close the producer object in the callback during failure. This will guarantee > in-order delivery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-938) KV-store API changes should be backward compatible to 0.10
[ https://issues.apache.org/jira/browse/SAMZA-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-938: --- Affects Version/s: 0.10.1 > KV-store API changes should be backward compatible to 0.10 > -- > > Key: SAMZA-938 > URL: https://issues.apache.org/jira/browse/SAMZA-938 > Project: Samza > Issue Type: Bug >Affects Versions: 0.10.1 >Reporter: Yi Pan (Data Infrastructure) > > SAMZA-813 introduced new methods in KeyValueStore and KeyValueIterator > interface classes, which in JDK7, breaks the backward compatibility for > customized implementation of KeyValueStore and KeyValueIterator. > As 0.10.1 is a bug-fix release, we should make sure that the backward > compatibility is retained. Hence, it would be better to have the extended > interface classes to implement the seekable functions, instead of changing > the existing classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SAMZA-938) KV-store API changes should be backward compatible to 0.10
Yi Pan (Data Infrastructure) created SAMZA-938: -- Summary: KV-store API changes should be backward compatible to 0.10 Key: SAMZA-938 URL: https://issues.apache.org/jira/browse/SAMZA-938 Project: Samza Issue Type: Bug Reporter: Yi Pan (Data Infrastructure) SAMZA-813 introduced new methods in KeyValueStore and KeyValueIterator interface classes, which in JDK7, breaks the backward compatibility for customized implementation of KeyValueStore and KeyValueIterator. As 0.10.1 is a bug-fix release, we should make sure that the backward compatibility is retained. Hence, it would be better to have the extended interface classes to implement the seekable functions, instead of changing the existing classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-924) Add disk space monitoring
[ https://issues.apache.org/jira/browse/SAMZA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-924: --- Fix Version/s: 0.10.1 > Add disk space monitoring > - > > Key: SAMZA-924 > URL: https://issues.apache.org/jira/browse/SAMZA-924 > Project: Samza > Issue Type: New Feature > Components: container >Reporter: Chris Pettitt >Assignee: Chris Pettitt > Fix For: 0.10.1 > > Attachments: baseline1.txt, baseline2.txt, experiment.txt, > rb45504.patch > > > This ticket tracks work to monitor the disk space used by stores in samza. It > is the initial step in a larger effort to restrict excessive disk usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-920) BrokerProxy.abdicateAll can get stuck on adding and removing the same partitions infinitely
[
https://issues.apache.org/jira/browse/SAMZA-920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Pan (Data Infrastructure) updated SAMZA-920:
---
Fix Version/s: 0.10.1
> BrokerProxy.abdicateAll can get stuck on adding and removing the same
> partitions infinitely
> ---
>
> Key: SAMZA-920
> URL: https://issues.apache.org/jira/browse/SAMZA-920
> Project: Samza
> Issue Type: Bug
> Components: kafka
>Affects Versions: 0.9.1, 0.10.1
>Reporter: Ivan Simoneko
>Assignee: Ivan Simoneko
> Labels: easyfix
> Fix For: 0.10.1
>
> Attachments: SAMZA-920_v1.patch
>
>
> abdicateAll is iterating over ConcurrentHashMap nextOffsets, removing and
> adding back elements which can result in an infinite iteration.
> {code}
> 2016-03-20 20:25:41,413 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
> [].
> 2016-03-20 20:25:41,414 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for:
> Map([] -> 20749911)
> 2016-03-20 20:25:41,414 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Adding new topic and partition
> [] to queue for
> 2016-03-20 20:25:41,414 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.GetOffset - Validating offset 20749911 for
> topic and partition []
> 2016-03-20 20:25:41,428 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.GetOffset - Able to successfully read from
> offset 20749911 for topic and partition []. Using it to
> instantiate consumer.
> 2016-03-20 20:25:41,428 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Got offset 20749911 for new topic
> and partition [].
> 2016-03-20 20:25:41,428 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Tried to start an already started
> broker proxy (BrokerProxy for :). Ignoring.
> 2016-03-20 20:25:41,429 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Claimed topic-partition
> ([]) for (BrokerProxy for :)
> 2016-03-20 20:25:41,429 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Removed []
> 2016-03-20 20:25:41,429 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
> [].
> 2016-03-20 20:25:41,429 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for:
> Map([] -> 20749909)
> 2016-03-20 20:25:41,429 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Adding new topic and partition
> [] to queue for
> 2016-03-20 20:25:41,429 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.GetOffset - Validating offset 20749909 for
> topic and partition []
> 2016-03-20 20:25:41,444 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.GetOffset - Able to successfully read from
> offset 20749909 for topic and partition []. Using it to
> instantiate consumer.
> 2016-03-20 20:25:41,444 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Got offset 20749909 for new topic
> and partition [].
> 2016-03-20 20:25:41,444 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Tried to start an already started
> broker proxy (BrokerProxy for :). Ignoring.
> 2016-03-20 20:25:41,444 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Claimed topic-partition
> ([]) for (BrokerProxy for :)
> 2016-03-20 20:25:41,444 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Removed []
> 2016-03-20 20:25:41,444 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
> [].
> 2016-03-20 20:25:41,444 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
>
[jira] [Updated] (SAMZA-920) BrokerProxy.abdicateAll can get stuck on adding and removing the same partitions infinitely
[
https://issues.apache.org/jira/browse/SAMZA-920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Pan (Data Infrastructure) updated SAMZA-920:
---
Assignee: Ivan Simoneko
> BrokerProxy.abdicateAll can get stuck on adding and removing the same
> partitions infinitely
> ---
>
> Key: SAMZA-920
> URL: https://issues.apache.org/jira/browse/SAMZA-920
> Project: Samza
> Issue Type: Bug
> Components: kafka
>Affects Versions: 0.9.1, 0.10.1
>Reporter: Ivan Simoneko
>Assignee: Ivan Simoneko
> Labels: easyfix
> Attachments: SAMZA-920_v1.patch
>
>
> abdicateAll is iterating over ConcurrentHashMap nextOffsets, removing and
> adding back elements which can result in an infinite iteration.
> {code}
> 2016-03-20 20:25:41,413 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
> [].
> 2016-03-20 20:25:41,414 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for:
> Map([] -> 20749911)
> 2016-03-20 20:25:41,414 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Adding new topic and partition
> [] to queue for
> 2016-03-20 20:25:41,414 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.GetOffset - Validating offset 20749911 for
> topic and partition []
> 2016-03-20 20:25:41,428 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.GetOffset - Able to successfully read from
> offset 20749911 for topic and partition []. Using it to
> instantiate consumer.
> 2016-03-20 20:25:41,428 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Got offset 20749911 for new topic
> and partition [].
> 2016-03-20 20:25:41,428 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Tried to start an already started
> broker proxy (BrokerProxy for :). Ignoring.
> 2016-03-20 20:25:41,429 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Claimed topic-partition
> ([]) for (BrokerProxy for :)
> 2016-03-20 20:25:41,429 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Removed []
> 2016-03-20 20:25:41,429 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
> [].
> 2016-03-20 20:25:41,429 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for:
> Map([] -> 20749909)
> 2016-03-20 20:25:41,429 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Adding new topic and partition
> [] to queue for
> 2016-03-20 20:25:41,429 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.GetOffset - Validating offset 20749909 for
> topic and partition []
> 2016-03-20 20:25:41,444 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.GetOffset - Able to successfully read from
> offset 20749909 for topic and partition []. Using it to
> instantiate consumer.
> 2016-03-20 20:25:41,444 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Got offset 20749909 for new topic
> and partition [].
> 2016-03-20 20:25:41,444 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Tried to start an already started
> broker proxy (BrokerProxy for :). Ignoring.
> 2016-03-20 20:25:41,444 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Claimed topic-partition
> ([]) for (BrokerProxy for :)
> 2016-03-20 20:25:41,444 DEBUG [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.BrokerProxy - Removed []
> 2016-03-20 20:25:41,444 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for
> [].
> 2016-03-20 20:25:41,444 INFO [SAMZA-BROKER-PROXY-BrokerProxy thread pointed
> at : for client ]
> org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing
[jira] [Commented] (SAMZA-906) Host Affinity - Minimize task reassignment when container count changes
[ https://issues.apache.org/jira/browse/SAMZA-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225374#comment-15225374 ] Yi Pan (Data Infrastructure) commented on SAMZA-906: Thanks! Merged and committed. > Host Affinity - Minimize task reassignment when container count changes > --- > > Key: SAMZA-906 > URL: https://issues.apache.org/jira/browse/SAMZA-906 > Project: Samza > Issue Type: Bug >Reporter: Jake Maes >Assignee: Jake Maes > Fix For: 0.10.1 > > Attachments: SAMZA-906.patch, SAMZA-906_2.patch, SAMZA-906_3.patch, > SAMZA-906_3.patch, SAMZA-906_4.patch, SAMZA-906_5.patch > > > When the job container count changes the default taskname grouper > (GroupByContainerCount) will reassign all the tasks among the new container > list in a round-robin fashion. This causes many of the tasks to shift to new > containers. The shifted tasks will be unable to restore state from local disk > because the new containers may not be assigned to the same host that the > task's original container was assigned to. > This ticket is to implement a task-to-container affinity, which complements > the container-to-host affinity in the current implementation. The > implementation will include a mapping which is persisted to the coordinator > stream and is used as the basis for the new task-to-container mapping > (ContainerModel). > If the container count doesn't change, the old task mapping will be used. > (Note that this will allow tools to inject custom mappings by writing to the > coordinator stream). > If the container count changes, a minimal number of tasks will be reassigned > from the persisted mapping, in order to "balance" the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SAMZA-919) Add milliseconds and thread name to log4j configs.
[ https://issues.apache.org/jira/browse/SAMZA-919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) resolved SAMZA-919. Resolution: Fixed +1ed on RBs. Merged and submitted. Thanks! > Add milliseconds and thread name to log4j configs. > --- > > Key: SAMZA-919 > URL: https://issues.apache.org/jira/browse/SAMZA-919 > Project: Samza > Issue Type: Bug >Reporter: Jake Maes >Assignee: Jake Maes > Fix For: 0.10.1 > > Attachments: SAMZA-919_hello-samza.patch, SAMZA-919_samza.patch > > > Although the Samza RunLoop is single-threaded, there are other threads used > in important areas. e.g. BrokerProxy. It's useful to log the thread name to > help reason about how the threads are interacting. > Also, the time stamps are currently logged with second granularity which can > make it hard to identify events that happen on sub-second time scale. > This ticket is to update the log4j configurations in the samza and > hello-samza repositories to include millisecond timestamp granularity and > threadname. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224539#comment-15224539 ] Yi Pan (Data Infrastructure) commented on SAMZA-727: Opened a JIRA to track the AM restart issue. SAMZA-929. > Support for Kerberos > > > Key: SAMZA-727 > URL: https://issues.apache.org/jira/browse/SAMZA-727 > Project: Samza > Issue Type: New Feature > Components: yarn >Affects Versions: 0.9.0 > Environment: YARN with Kerberos >Reporter: Qi FU >Assignee: Chen Song > Attachments: SAMZA-727.patch > > > Samza doesn't support Kerberos, which is very common for YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SAMZA-929) Support AM restart in Kerberos secured YARN after the original token in launch context expires
Yi Pan (Data Infrastructure) created SAMZA-929: -- Summary: Support AM restart in Kerberos secured YARN after the original token in launch context expires Key: SAMZA-929 URL: https://issues.apache.org/jira/browse/SAMZA-929 Project: Samza Issue Type: Sub-task Reporter: Yi Pan (Data Infrastructure) In Kerberos secured YARN cluster, an initial token is given in the launch context to RM for a job. When that token expires, RM will not be able to restart failed AM. As for now, we will have to re-submit the job to refresh the token in the launch context. This ticket is searching for a solution that can refresh the token in the launch context for AM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224526#comment-15224526 ] Yi Pan (Data Infrastructure) commented on SAMZA-727: [~steve_l], thanks a lot for replying. As for point 2, I think that for now, it is OK if AM fail to restart after the original token in launch context expires. We will have to re-submit the job in this case. [~capricornius], based on the above discussion. I think that we can go w/ your design for now. Let's open another JIRA regarding to "support AM restart after the original Kerberos token in launch context expires". > Support for Kerberos > > > Key: SAMZA-727 > URL: https://issues.apache.org/jira/browse/SAMZA-727 > Project: Samza > Issue Type: New Feature > Components: yarn >Affects Versions: 0.9.0 > Environment: YARN with Kerberos >Reporter: Qi FU >Assignee: Chen Song > Attachments: SAMZA-727.patch > > > Samza doesn't support Kerberos, which is very common for YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-921) Consolidate LocalityManager and TaskAssignmentManager
[ https://issues.apache.org/jira/browse/SAMZA-921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222130#comment-15222130 ] Yi Pan (Data Infrastructure) commented on SAMZA-921: Hi, [~jmaes], yeah, your description is accurate. IMHO, we need to have the per task to host message anyways. Hence, using the single message to carry task->container-host mapping actually reduce the number of messages in the CoordinatorStream, even after compaction. Besides that, the repetition of container in each message should not be a huge cost, since a) it only adds a field in the value, not the whole message; b) the number of tasks in a single container usually does not go beyond 10, which results in a bounded overhead. > Consolidate LocalityManager and TaskAssignmentManager > - > > Key: SAMZA-921 > URL: https://issues.apache.org/jira/browse/SAMZA-921 > Project: Samza > Issue Type: Bug >Reporter: Jake Maes > > As part of the work and discussion around SAMZA-906 there were a couple > observations about how locality information should be managed in Samza. > 1. There should be one locality manager that is ultimately responsible for > mapping tasks to hosts. To do this, it may also need to manage separate > intermediate mappings from task->container and container->host, though there > are some contexts where both mappings will be needed. (e.g. some > implementations of BalancingTaskNameGrouper) > 2. Locality information should be written centrally. This facilitates a > broadcast-like system, in which one leader writes the coordinator stream once > and that information is consumed by all non-leaders either directly or > through the leader. This also has the advantage that the leader can better > track changes like a decrease in container count and clean up the > container->host mapping, whereas the containers cannot do this naturally. > Why doesn't it already work this way? > 1. The locality manager writes the locality in the container in order to > ensure that the task was actually executing before writing the locality to > the coordinator stream. This prevents a scenario where a container is > attempted and fails on multiple hosts, thrashing the locality information and > losing the local state. However, after SAMZA-871 is implemented, the leader > will have the means to determine that the container is running, which will > enable the locality to be written centrally. > 2. Since the container->host mapping is written in the containers, it > registers itself as a CoordinatorStreamManager with "source" specific to the > container doing the writing. Conversely, the TaskAssignmentManager writes all > mappings centrally, so it uses one "TaskAssignmentManager" source. If, > however the locality manager were updated to also write centrally, they could > be combined and register with one "LocalityManager" source, which enables > consolidation of the currently-separate CoordinatorStreamManager > implementations (LocalityManager and TaskAssignmentManager) > So, after SAMZA-871 we should be able to combine the LocalityManager and > TaskAssignmentManager into one class that writes everything centrally, but > still preserves the guarantee that the container-host mapping is not written > until the container is actually running. That work is the goal of this task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-925) Explicit restart containers to pick up dynamic JobModel changes
[ https://issues.apache.org/jira/browse/SAMZA-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220319#comment-15220319 ] Yi Pan (Data Infrastructure) commented on SAMZA-925: [~cpsoman], this may be something that you are interested as well. > Explicit restart containers to pick up dynamic JobModel changes > --- > > Key: SAMZA-925 > URL: https://issues.apache.org/jira/browse/SAMZA-925 > Project: Samza > Issue Type: Sub-task >Reporter: Yi Pan (Data Infrastructure) >Assignee: Alex Buck > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-348) Configure Samza jobs through a stream
[ https://issues.apache.org/jira/browse/SAMZA-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220302#comment-15220302 ] Yi Pan (Data Infrastructure) commented on SAMZA-348: Hi, [~alex.buck10], I have created the sub-task SAMZA-921 and assigned to you. There have been many updates regarding to the CoordinatorStream and the JobCoordinator recently. Please check SAMZA-448 for the implementation of CoordinatorStream. This might also be related to the refactoring of JobCoordinator that we are actively working on: SAMZA-881. So, it would be good if the design/implementation of the dynamic re-config via restart can be compatible w/ the refactored JobCoordinator as well. As for the bugs you found in JSON deserialization, feel free to open a JIRA. Everyone should have the power to open JIRA in Samza. Thanks a lot! > Configure Samza jobs through a stream > - > > Key: SAMZA-348 > URL: https://issues.apache.org/jira/browse/SAMZA-348 > Project: Samza > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Chris Riccomini >Assignee: Chris Riccomini > Labels: design, project > Attachments: DESIGN-SAMZA-348-0.md, DESIGN-SAMZA-348-0.pdf, > DESIGN-SAMZA-348-1.md, DESIGN-SAMZA-348-1.pdf > > > Samza's existing config setup is problematic for a number of reasons: > # It's completely immutable once a job starts. This prevents any dynamic > reconfiguration and auto-scaling. It is debatable whether we want these > feature or not, but our existing implementation actively prevents it. See > SAMZA-334 for discussion. > # We pass existing configuration through environment variables. YARN exports > environment variables in a shell script, which limits the size to the varargs > length on the machine. This is usually ~128KB. See SAMZA-333 and SAMZA-337 > for details. > # User-defined configuration (the Config object) and programmatic > configuration (checkpoints and TaskName:State mappings (see SAMZA-123)) are > handled differently. It's debatable whether this makes sense. > In SAMZA-123, [~jghoman] and I propose implementing a ConfigLog. This log > would replace both the checkpoint topic and the existing config environment > variables in SamzaContainer and Samza's YARN AM. > I'd like to keep this ticket's scope limited to just the implementation of > the ConfigLog, and not re-designing how Samza's config is used in the code > (SAMZA-40). We should, however, discuss how this feature would affect dynamic > reconfiguration/auto-scaling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-925) Explicit restart containers to pick up dynamic JobModel changes
[ https://issues.apache.org/jira/browse/SAMZA-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-925: --- Assignee: Alex Buck (was: Yi Pan (Data Infrastructure)) > Explicit restart containers to pick up dynamic JobModel changes > --- > > Key: SAMZA-925 > URL: https://issues.apache.org/jira/browse/SAMZA-925 > Project: Samza > Issue Type: Sub-task >Reporter: Yi Pan (Data Infrastructure) >Assignee: Alex Buck > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (SAMZA-925) Explicit restart containers to pick up dynamic JobModel changes
[ https://issues.apache.org/jira/browse/SAMZA-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) reassigned SAMZA-925: -- Assignee: Yi Pan (Data Infrastructure) > Explicit restart containers to pick up dynamic JobModel changes > --- > > Key: SAMZA-925 > URL: https://issues.apache.org/jira/browse/SAMZA-925 > Project: Samza > Issue Type: Sub-task >Reporter: Yi Pan (Data Infrastructure) >Assignee: Yi Pan (Data Infrastructure) > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SAMZA-925) Explicit restart containers to pick up dynamic JobModel changes
Yi Pan (Data Infrastructure) created SAMZA-925: -- Summary: Explicit restart containers to pick up dynamic JobModel changes Key: SAMZA-925 URL: https://issues.apache.org/jira/browse/SAMZA-925 Project: Samza Issue Type: Sub-task Reporter: Yi Pan (Data Infrastructure) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-913) CoordinatorStreamSystemConsumer drops messages when they are considered equivalent
[
https://issues.apache.org/jira/browse/SAMZA-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15212351#comment-15212351
]
Yi Pan (Data Infrastructure) commented on SAMZA-913:
Merged and submitted. Thanks!
> CoordinatorStreamSystemConsumer drops messages when they are considered
> equivalent
> --
>
> Key: SAMZA-913
> URL: https://issues.apache.org/jira/browse/SAMZA-913
> Project: Samza
> Issue Type: Bug
>Reporter: Jake Maes
>Assignee: Jake Maes
> Fix For: 0.10.1
>
> Attachments: SAMZA-913.patch
>
>
> When CoordinatorStreamSystemConsumer bootstraps, it adds the messages to a
> LinkedHashSet ("bootstrappedStreamSet"). The intent seems to be:
> 1. Messages will be processed in the order they were consumed.
> 2. Only the latest copy of a message will be stored.
> That second assumption turns out to be false with the current implementation.
> In Java, Set.add() only adds an element if it doesn't already exist in the
> Set. Further, CoordinatorStreamMessage.equals() relies on the key set and
> values, but not the message offset or timestamp, so the following set of
> messages could occur:
> key1 -> value1 // added to bootstrappedStreamSet
> key1 -> value2 // added to bootstrappedStreamSet
> key1 -> value1 // duplicate to first message, not added
> Thus the final state will be (incorrectly):
> key1 -> value2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SAMZA-914) Implement a composable operator APIs for direct programming of DAG in Java
[ https://issues.apache.org/jira/browse/SAMZA-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-914: --- Description: In the process of design/refining the physical operator layer for SAMZA-552, there is a need to design/implement a set of fluent programming APIs s.t. users may be able to directly write Java programs using the physical operators. The programming APIs would serve as an intermediate layer that serves the following purposes: # Users can choose to directly use physical operators (like window/join/filter etc.) in their Java code # The SQL parser and engine can use this API to construct the physical operators and the execution DAG. was: In the process of design/refining the physical operator layer for SAMZA-552, there is a need to design/implement a fluent programming APIs s.t. users may be able to directly write Java programs using the physical operators. The programming APIs would serve as an intermediate layer that serves the following purposes: # Users can choose to directly use physical operators (like window/join/filter etc.) in their Java code # The SQL parser and engine can use this API to construct the physical operators and the execution DAG. > Implement a composable operator APIs for direct programming of DAG in Java > -- > > Key: SAMZA-914 > URL: https://issues.apache.org/jira/browse/SAMZA-914 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) >Assignee: Yi Pan (Data Infrastructure) > > In the process of design/refining the physical operator layer for SAMZA-552, > there is a need to design/implement a set of fluent programming APIs s.t. > users may be able to directly write Java programs using the physical > operators. > The programming APIs would serve as an intermediate layer that serves the > following purposes: > # Users can choose to directly use physical operators (like > window/join/filter etc.) in their Java code > # The SQL parser and engine can use this API to construct the physical > operators and the execution DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-914) Implement a composable operator APIs for direct programming of DAG in Java
[ https://issues.apache.org/jira/browse/SAMZA-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1527#comment-1527 ] Yi Pan (Data Infrastructure) commented on SAMZA-914: Ah... sorry I forgot that. It is public now. Thanks! > Implement a composable operator APIs for direct programming of DAG in Java > -- > > Key: SAMZA-914 > URL: https://issues.apache.org/jira/browse/SAMZA-914 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) >Assignee: Yi Pan (Data Infrastructure) > > In the process of design/refining the physical operator layer for SAMZA-552, > there is a need to design/implement a fluent programming APIs s.t. users may > be able to directly write Java programs using the physical operators. > The programming APIs would serve as an intermediate layer that serves the > following purposes: > # Users can choose to directly use physical operators (like > window/join/filter etc.) in their Java code > # The SQL parser and engine can use this API to construct the physical > operators and the execution DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-914) Implement a composable operator APIs for direct programming of DAG in Java
[ https://issues.apache.org/jira/browse/SAMZA-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211072#comment-15211072 ] Yi Pan (Data Infrastructure) commented on SAMZA-914: The initial draft of operator programming APIs (stubs) and example application code are [here|https://reviews.apache.org/r/45324/] > Implement a composable operator APIs for direct programming of DAG in Java > -- > > Key: SAMZA-914 > URL: https://issues.apache.org/jira/browse/SAMZA-914 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) >Assignee: Yi Pan (Data Infrastructure) > > In the process of design/refining the physical operator layer for SAMZA-552, > there is a need to design/implement a fluent programming APIs s.t. users may > be able to directly write Java programs using the physical operators. > The programming APIs would serve as an intermediate layer that serves the > following purposes: > # Users can choose to directly use physical operators (like > window/join/filter etc.) in their Java code > # The SQL parser and engine can use this API to construct the physical > operators and the execution DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (SAMZA-914) Implement a composable operator APIs for direct programming of DAG in Java
[ https://issues.apache.org/jira/browse/SAMZA-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) reassigned SAMZA-914: -- Assignee: Yi Pan (Data Infrastructure) > Implement a composable operator APIs for direct programming of DAG in Java > -- > > Key: SAMZA-914 > URL: https://issues.apache.org/jira/browse/SAMZA-914 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) >Assignee: Yi Pan (Data Infrastructure) > > In the process of design/refining the physical operator layer for SAMZA-552, > there is a need to design/implement a fluent programming APIs s.t. users may > be able to directly write Java programs using the physical operators. > The programming APIs would serve as an intermediate layer that serves the > following purposes: > # Users can choose to directly use physical operators (like > window/join/filter etc.) in their Java code > # The SQL parser and engine can use this API to construct the physical > operators and the execution DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SAMZA-916) Design and implement operator programing APIs to support split and join
Yi Pan (Data Infrastructure) created SAMZA-916:
--
Summary: Design and implement operator programing APIs to support
split and join
Key: SAMZA-916
URL: https://issues.apache.org/jira/browse/SAMZA-916
Project: Samza
Issue Type: Sub-task
Reporter: Yi Pan (Data Infrastructure)
This is to implement programming APIs to support non-linear DAG of operators.
The key operators to support is to support split and join operations.
Some code example would be:
{code}
commonSource = sourceStream.map(mapper);
commonSource.filter(filter1).join(joiner1).flatMap(flatMapper1).sink(sink);
commonSource.filter(filter2).flatMap(flatMapper2).join(joiner1);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SAMZA-914) Implement a composable operator APIs for direct programming of DAG in Java
[ https://issues.apache.org/jira/browse/SAMZA-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-914: --- Summary: Implement a composable operator APIs for direct programming of DAG in Java (was: Implement a composable operator APIs for direct programming in Java) > Implement a composable operator APIs for direct programming of DAG in Java > -- > > Key: SAMZA-914 > URL: https://issues.apache.org/jira/browse/SAMZA-914 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) > > In the process of design/refining the physical operator layer for SAMZA-552, > there is a need to design/implement a fluent programming APIs s.t. users may > be able to directly write Java programs using the physical operators. > The programming APIs would serve as an intermediate layer that serves the > following purposes: > # Users can choose to directly use physical operators (like > window/join/filter etc.) in their Java code > # The SQL parser and engine can use this API to construct the physical > operators and the execution DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SAMZA-915) Design and implement operator programming APIs to support linear pipeline
Yi Pan (Data Infrastructure) created SAMZA-915:
--
Summary: Design and implement operator programming APIs to support
linear pipeline
Key: SAMZA-915
URL: https://issues.apache.org/jira/browse/SAMZA-915
Project: Samza
Issue Type: Sub-task
Reporter: Yi Pan (Data Infrastructure)
This would include the programming APIs that support the following linear
pipeline:
{code}
sourceStream.filter(filter).map(mapper).flatMap(flatMapper).sink(sink);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SAMZA-907) Make samza-sql-core and samza-sql-calcite as optionally compiled modules w/ JDK8
[ https://issues.apache.org/jira/browse/SAMZA-907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210698#comment-15210698 ] Yi Pan (Data Infrastructure) commented on SAMZA-907: Got +1 on RB. Merged and committed. Thanks! > Make samza-sql-core and samza-sql-calcite as optionally compiled modules w/ > JDK8 > > > Key: SAMZA-907 > URL: https://issues.apache.org/jira/browse/SAMZA-907 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) >Assignee: Yi Pan (Data Infrastructure) > Attachments: SAMZA-907-0.patch > > > To enable JDK8 source level features in the new modules, we can optionally > compile the modules that uses JDK8 new source level features when JDK8 > compile option is turned on. > This will accelerate the design of stream operator APIs (i.e. SAMZA-552) a > lot by leveraging new features in JDK8, like stream and lambda expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-907) Make samza-sql-core and samza-sql-calcite as optionally compiled modules w/ JDK8
[ https://issues.apache.org/jira/browse/SAMZA-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-907: --- Attachment: SAMZA-907-0.patch > Make samza-sql-core and samza-sql-calcite as optionally compiled modules w/ > JDK8 > > > Key: SAMZA-907 > URL: https://issues.apache.org/jira/browse/SAMZA-907 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) >Assignee: Yi Pan (Data Infrastructure) > Attachments: SAMZA-907-0.patch > > > To enable JDK8 source level features in the new modules, we can optionally > compile the modules that uses JDK8 new source level features when JDK8 > compile option is turned on. > This will accelerate the design of stream operator APIs (i.e. SAMZA-552) a > lot by leveraging new features in JDK8, like stream and lambda expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-907) Make samza-sql-core and samza-sql-calcite as optionally compiled modules w/ JDK8
[ https://issues.apache.org/jira/browse/SAMZA-907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209045#comment-15209045 ] Yi Pan (Data Infrastructure) commented on SAMZA-907: RB: https://reviews.apache.org/r/45246/ > Make samza-sql-core and samza-sql-calcite as optionally compiled modules w/ > JDK8 > > > Key: SAMZA-907 > URL: https://issues.apache.org/jira/browse/SAMZA-907 > Project: Samza > Issue Type: Bug >Reporter: Yi Pan (Data Infrastructure) >Assignee: Yi Pan (Data Infrastructure) > > To enable JDK8 source level features in the new modules, we can optionally > compile the modules that uses JDK8 new source level features when JDK8 > compile option is turned on. > This will accelerate the design of stream operator APIs (i.e. SAMZA-552) a > lot by leveraging new features in JDK8, like stream and lambda expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-908) Fix dependency issues in samza-sql-calcite
[
https://issues.apache.org/jira/browse/SAMZA-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209051#comment-15209051
]
Yi Pan (Data Infrastructure) commented on SAMZA-908:
Merged and submitted to samza-sql branch.
> Fix dependency issues in samza-sql-calcite
> --
>
> Key: SAMZA-908
> URL: https://issues.apache.org/jira/browse/SAMZA-908
> Project: Samza
> Issue Type: Bug
> Components: sql
>Affects Versions: 0.10.0
>Reporter: Yi Pan (Data Infrastructure)
>Assignee: Yi Pan (Data Infrastructure)
>
> It seems that some indirect dependencies required by calcite-core are missing
> from maven central now.
> {noformat}
> Could not resolve all dependencies for configuration
> ':samza-sql-calcite:compile'.
> > Could not find org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
> Searched in the following locations:
>
> http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
>
> http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
>
> file:/home/yipan/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
>
> file:/home/yipan/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
> Required by:
> org.apache.samza:samza-sql-calcite:0.10.1-SNAPSHOT >
> org.apache.calcite:calcite-core:1.5.0
> {noformat}
> It seems that the missing package is available from http://conjars.org/repo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SAMZA-908) Fix dependency issues in samza-sql-calcite
[
https://issues.apache.org/jira/browse/SAMZA-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Pan (Data Infrastructure) updated SAMZA-908:
---
Component/s: sql
> Fix dependency issues in samza-sql-calcite
> --
>
> Key: SAMZA-908
> URL: https://issues.apache.org/jira/browse/SAMZA-908
> Project: Samza
> Issue Type: Bug
> Components: sql
>Affects Versions: 0.10.0
>Reporter: Yi Pan (Data Infrastructure)
>Assignee: Yi Pan (Data Infrastructure)
>
> It seems that some indirect dependencies required by calcite-core are missing
> from maven central now.
> {noformat}
> Could not resolve all dependencies for configuration
> ':samza-sql-calcite:compile'.
> > Could not find org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
> Searched in the following locations:
>
> http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
>
> http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
>
> file:/home/yipan/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
>
> file:/home/yipan/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
> Required by:
> org.apache.samza:samza-sql-calcite:0.10.1-SNAPSHOT >
> org.apache.calcite:calcite-core:1.5.0
> {noformat}
> It seems that the missing package is available from http://conjars.org/repo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SAMZA-908) Fix dependency issues in samza-sql-calcite
[
https://issues.apache.org/jira/browse/SAMZA-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207575#comment-15207575
]
Yi Pan (Data Infrastructure) commented on SAMZA-908:
The patch is simple. Hence, directly posting here w/o attachment:
{noformat}
diff --git a/build.gradle b/build.gradle
index 7faa492..685b24d 100644
--- a/build.gradle
+++ b/build.gradle
@@ -352,6 +352,12 @@ project(":samza-sql-core") {
project(":samza-sql-calcite") {
apply plugin: 'java'
+ repositories {
+maven {
+ url "http://conjars.org/repo;
+}
+ }
+
dependencies {
compile project(":samza-sql-core")
compile "org.apache.calcite:calcite-core:$calciteVersion"
{noformat}
> Fix dependency issues in samza-sql-calcite
> --
>
> Key: SAMZA-908
> URL: https://issues.apache.org/jira/browse/SAMZA-908
> Project: Samza
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Yi Pan (Data Infrastructure)
>
> It seems that some indirect dependencies required by calcite-core are missing
> from maven central now.
> {noformat}
> Could not resolve all dependencies for configuration
> ':samza-sql-calcite:compile'.
> > Could not find org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
> Searched in the following locations:
>
> http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
>
> http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
>
> file:/home/yipan/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
>
> file:/home/yipan/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
> Required by:
> org.apache.samza:samza-sql-calcite:0.10.1-SNAPSHOT >
> org.apache.calcite:calcite-core:1.5.0
> {noformat}
> It seems that the missing package is available from http://conjars.org/repo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SAMZA-908) Fix dependency issues in samza-sql-calcite
[
https://issues.apache.org/jira/browse/SAMZA-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207573#comment-15207573
]
Yi Pan (Data Infrastructure) commented on SAMZA-908:
RB available: https://reviews.apache.org/r/45189/
> Fix dependency issues in samza-sql-calcite
> --
>
> Key: SAMZA-908
> URL: https://issues.apache.org/jira/browse/SAMZA-908
> Project: Samza
> Issue Type: Bug
>Reporter: Yi Pan (Data Infrastructure)
>
> It seems that some indirect dependencies required by calcite-core are missing
> from maven central now.
> {noformat}
> Could not resolve all dependencies for configuration
> ':samza-sql-calcite:compile'.
> > Could not find org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
> Searched in the following locations:
>
> http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
>
> http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
>
> file:/home/yipan/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
>
> file:/home/yipan/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
> Required by:
> org.apache.samza:samza-sql-calcite:0.10.1-SNAPSHOT >
> org.apache.calcite:calcite-core:1.5.0
> {noformat}
> It seems that the missing package is available from http://conjars.org/repo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SAMZA-893) Fix a bug with host affinity request expiration introduced in SAMZA-867
[ https://issues.apache.org/jira/browse/SAMZA-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207565#comment-15207565 ] Yi Pan (Data Infrastructure) commented on SAMZA-893: [~jmaes], thanks for splitting the patches. I would recommend to open a separate JIRA for the unit test fix, s.t. we can commit and close on this one. +1 on the logic fix already. > Fix a bug with host affinity request expiration introduced in SAMZA-867 > --- > > Key: SAMZA-893 > URL: https://issues.apache.org/jira/browse/SAMZA-893 > Project: Samza > Issue Type: Bug >Reporter: Jake Maes >Assignee: Jake Maes > Attachments: SAMZA-893.patch, SAMZA-893_logic.patch > > > The expiration logic change in the patch for SAMZA-867 was wrong. It > simplified the conditional, but would cause expired requests to never get > rescheduled on ANY_HOST. > This ticket is to fix that logic and the unit tests, which didn't fail after > the change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SAMZA-909) Merge samza-sql branch to master
Yi Pan (Data Infrastructure) created SAMZA-909: -- Summary: Merge samza-sql branch to master Key: SAMZA-909 URL: https://issues.apache.org/jira/browse/SAMZA-909 Project: Samza Issue Type: Bug Reporter: Yi Pan (Data Infrastructure) Assignee: Yi Pan (Data Infrastructure) With SAMZA-907 that allows skipping the samza-sql modules in JDK7 builds (i.e. which is the official release build), we can safely merge the samza-sql experimental modules in master w/o impact the released jars. It will help to keep all the active projects in the trunk development mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SAMZA-908) Fix dependency issues in samza-sql-calcite
Yi Pan (Data Infrastructure) created SAMZA-908:
--
Summary: Fix dependency issues in samza-sql-calcite
Key: SAMZA-908
URL: https://issues.apache.org/jira/browse/SAMZA-908
Project: Samza
Issue Type: Bug
Reporter: Yi Pan (Data Infrastructure)
It seems that some indirect dependencies required by calcite-core are missing
from maven central now.
{noformat}
Could not resolve all dependencies for configuration
':samza-sql-calcite:compile'.
> Could not find org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
Searched in the following locations:
http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
file:/home/yipan/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
file:/home/yipan/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
Required by:
org.apache.samza:samza-sql-calcite:0.10.1-SNAPSHOT >
org.apache.calcite:calcite-core:1.5.0
{noformat}
It seems that the missing package is available from http://conjars.org/repo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SAMZA-907) Make samza-sql-core and samza-sql-calcite as optionally compiled modules w/ JDK8
Yi Pan (Data Infrastructure) created SAMZA-907: -- Summary: Make samza-sql-core and samza-sql-calcite as optionally compiled modules w/ JDK8 Key: SAMZA-907 URL: https://issues.apache.org/jira/browse/SAMZA-907 Project: Samza Issue Type: Bug Reporter: Yi Pan (Data Infrastructure) Assignee: Yi Pan (Data Infrastructure) To enable JDK8 source level features in the new modules, we can optionally compile the modules that uses JDK8 new source level features when JDK8 compile option is turned on. This will accelerate the design of stream operator APIs (i.e. SAMZA-552) a lot by leveraging new features in JDK8, like stream and lambda expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-813) Add Seek functionality to KeyValueStoreIterator
[ https://issues.apache.org/jira/browse/SAMZA-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183374#comment-15183374 ] Yi Pan (Data Infrastructure) commented on SAMZA-813: [~amitra123], I added one comment on the [RB|https://reviews.apache.org/r/41068/]. Thanks! > Add Seek functionality to KeyValueStoreIterator > --- > > Key: SAMZA-813 > URL: https://issues.apache.org/jira/browse/SAMZA-813 > Project: Samza > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Amit Yadav >Assignee: Amit Yadav > Attachments: SAMZA-813.0.patch > > > The KeyValueIterator today does not support seeking to a particular key. In > some scenarios, seeking to a key provides huge benefits (E.g when there are > large number of deletes which causes issues- > https://www.facebook.com/groups/rocksdb.dev/permalink/604723469626171/) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-205) Bring Hello Samza in as an integration test
[ https://issues.apache.org/jira/browse/SAMZA-205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181426#comment-15181426 ] Yi Pan (Data Infrastructure) commented on SAMZA-205: [~yatessb], sorry to chime in late. I would think that if we bring in hs as an integration test, the main purpose is to test that every check-in to Samza master branch would work well w/ Hello-Samza latest branch. I think that it would be easier to first support the trunk head development issue. As part of the release process, we need to merge latest to master branch and validate hs works. But we can have a separate solution to support release validation, which should validate Hello-Samza master branch w/ a released branch in Samza. > Bring Hello Samza in as an integration test > --- > > Key: SAMZA-205 > URL: https://issues.apache.org/jira/browse/SAMZA-205 > Project: Samza > Issue Type: Task > Components: hello-samza, test >Reporter: Jakob Homan >Assignee: Steve Yates > > Hello Samza has proven to be a good integration test - ie, we break it on a > regular basis - but we don't realize it until someone manually runs HS > against the latest jars. It would be better to have a gradle task that > checks out the code, replaces the jars, tries to start up and runs for a bit. > > Because HS requires internet access, we shouldn't have this run every time, > but just as an extra step committers can use before actually checking in code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-854) Upgrade Zookeeper dependency to 3.4.3
[ https://issues.apache.org/jira/browse/SAMZA-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181413#comment-15181413 ] Yi Pan (Data Infrastructure) commented on SAMZA-854: [~navina], is there any specific reason that we stick w/ 3.4.3? I believe that 3.4.5+ was recommended by ZooKeeper community as a stable version. > Upgrade Zookeeper dependency to 3.4.3 > - > > Key: SAMZA-854 > URL: https://issues.apache.org/jira/browse/SAMZA-854 > Project: Samza > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Navina Ramesh > Fix For: 0.10.1 > > > For a long time now, we have been using 3.3.4 version of zookeeper. We should > upgrade to 3.4.3. > This will also involve changing the dependencies in hello-samza setup files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-881) Re-think the Samza Job Coordinator
[ https://issues.apache.org/jira/browse/SAMZA-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181405#comment-15181405 ] Yi Pan (Data Infrastructure) commented on SAMZA-881: Hi, [~jagadish1...@gmail.com], the proposal looks good to me. I am attaching a few comments below: # When we describe mutual exclusiveness in partition assignment, please exclude broadcast stream in the discussion # Be consistent w/ terms: in page 2, “leader container” vs “leader process” # In the architecture graph, it would be nice to label the text on the edges w/ execution order # Where is the container liveness management module in the design of JobCoordinator? W/ SAMZA-871, requesting for direct heart beat between the containers and AM (i.e. essentially followers and leaders in the new design), I think that we should have a separate pluggable module for this, in addition to ContainerProcessManager, which is just interface to allocate/request processes. # Case 3.2 is the prototype implemented in SAMZA-516, right? We should call it out. Thanks a lot for the design! > Re-think the Samza Job Coordinator > -- > > Key: SAMZA-881 > URL: https://issues.apache.org/jira/browse/SAMZA-881 > Project: Samza > Issue Type: Improvement >Reporter: Jagadish >Assignee: Jagadish > Attachments: SamzaJobCoordinatorRe-designProposal.pdf > > > Currently, the only way to run Samza containers in distributed mode is using > Yarn. However, there has been interest to run Samza on top of other resource > managers with the recent explosion in the # of such systems. Users have also > requested us to run Samza as a library, and to run Samza on Docker containers > managed by Kubernetes. > We must re-think the JobCoordinator functionality as follows: > 1. ID assignment: Provide an ID to each SamzaContainer. > 2. JobModel agreement: Ensure containers agree on a JobModel. > 3. Re-start the SamzaContainer when the job model changes. > This will arguably require some leader election (depending on how users > choose to run Samza). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181373#comment-15181373 ] Yi Pan (Data Infrastructure) commented on SAMZA-727: Hi, [~steve_l], just noticed one thing: # if the original HDFS delegation token in launch context can not be refreshed, and AM tries to request a new container to be allocated and launched, the NodeManager would get the original HDFS delegation token which may have expired. That will cause the container fails to launch, right? If that is true, we still have not solved the issue to allow: ## Containers can be launched even after the original HDFS token in the launch context expires Is it possible for AM to pass around the new delegation token via NMClient? > Support for Kerberos > > > Key: SAMZA-727 > URL: https://issues.apache.org/jira/browse/SAMZA-727 > Project: Samza > Issue Type: New Feature > Components: yarn >Affects Versions: 0.9.0 > Environment: YARN with Kerberos >Reporter: Qi FU >Assignee: Chen Song > Fix For: 0.10.1 > > Attachments: SAMZA-727.patch > > > Samza doesn't support Kerberos, which is very common for YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180637#comment-15180637 ] Yi Pan (Data Infrastructure) commented on SAMZA-727: [~steve_l], thanks for clarification! Appreciate it! [~capricornius], based on the above, I believe that if AM also refresh the HDFS delegation token and distribute it to containers, as long as AM is still alive, containers will be able to restart. The only problem is now that the initial HDFS delegation token in the launch context can not be refreshed in RM and when AM dies after that token expires, job will not be successfully re-launched. I think that it should be acceptable for now. Thanks! > Support for Kerberos > > > Key: SAMZA-727 > URL: https://issues.apache.org/jira/browse/SAMZA-727 > Project: Samza > Issue Type: New Feature > Components: yarn >Affects Versions: 0.9.0 > Environment: YARN with Kerberos >Reporter: Qi FU >Assignee: Chen Song > Fix For: 0.10.1 > > Attachments: SAMZA-727.patch > > > Samza doesn't support Kerberos, which is very common for YARN cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-863) Support multi-threading in samza tasks
[ https://issues.apache.org/jira/browse/SAMZA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180577#comment-15180577 ] Yi Pan (Data Infrastructure) commented on SAMZA-863: [~xinyu], thanks for the design doc and sorry for the late review. It looks great to me. I have just a few additional points, given that [~cpettitt-linkedin] and you have already discussed a lot: # What’s the sequence of wrappedTask.init() and ThreadedStreamTask.init()? I assume that ThreadedStreamTask is not public API which we will control the implementation of its init()? # What’s the user code vs what’s Samza framework code? Is the implementation of ThreadedStreamTask and ParSeqStreamTask being “reference implementation” of user code? Or the interface AsyncStreamTask is more of SPI instead of API and framework developer should implement it as a service/lib to be used by application developers? # As for the callbacks not triggered issue, I think that we should implement the timeout mechanism in the TaskCallback. Depending on whether the implementation enforces in-order execution of the callbacks, the failure of the first callback may or may not trigger the failure of all pending callbacks. This should be implemented in SamzaContainer or TaskInstance class as well. # For the race condition between window and process as [~cpettitt-linkedin] pointed out, I think to retain the current semantic may be critical for existing users. If user choose to implement WindowableTask interface, we should make sure that all pending process() are done before we invoke user implemented window() method. With the assumption that a) the window() method is not invoked often; b) timeout on callbacks will not block window() invocation forever, I think that would be a reasonable solution. Also, do we see any need for AsyncWindowableTask? We probably can mentioned it as "extra features" not in the first design. Thanks! > Support multi-threading in samza tasks > -- > > Key: SAMZA-863 > URL: https://issues.apache.org/jira/browse/SAMZA-863 > Project: Samza > Issue Type: New Feature >Reporter: Xinyu Liu >Assignee: Xinyu Liu > Attachments: DESIGN-SAMZA-863-0.pdf, DESIGN-SAMZA-863-1.pdf > > > Currently a samza container executes the tasks sequentially in a single > thread. For example, we have message 1 and 2 in the pending queue for task 1 > and task 2. Task 1 will process message 1, and until its completion task 2 > can process message 2. If we want to handle more messages in parallel, we > have to increase the container count, e.g. from 1 to 2 in the example. > While this solution has been working for many CPU-bound job scenarios, we do > see its drawback for IO-bound jobs.In this kind of jobs, the task makes > IO/Network requests, i.e, db calls, rest calls or external service RPC calls. > These IO calls significantly slow down the task processing. We can increase > container number in order to parallelize the IO calls, but it results in low > CPU utilization. If we can improve CPU utilization by allocating multiple > contains in the same CPU core, it will still cause dramatic memory growth due > to the memory being allocated for each container. > To better scale the performance of IO-bound jobs, we are proposing to support > multi-threaded processing in samza. The design proposal will come soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-884) Upgrade Jackson to 1.9.13 in hello-samza
[ https://issues.apache.org/jira/browse/SAMZA-884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180442#comment-15180442 ] Yi Pan (Data Infrastructure) commented on SAMZA-884: Got +1 on RB. Merged and submitted. Thanks! > Upgrade Jackson to 1.9.13 in hello-samza > > > Key: SAMZA-884 > URL: https://issues.apache.org/jira/browse/SAMZA-884 > Project: Samza > Issue Type: Bug > Components: hello-samza >Reporter: Navina Ramesh >Assignee: Yi Pan (Data Infrastructure) > Labels: newbie > Fix For: 0.10.1 > > Attachments: SAMZA-884-0.patch > > > As a result of jackson version upgrade in SAMZA-878 , the hello-samza job > breaks. We need to change the dependency there as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-884) Upgrade Jackson to 1.9.13 in hello-samza
[ https://issues.apache.org/jira/browse/SAMZA-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-884: --- Attachment: SAMZA-884-0.patch > Upgrade Jackson to 1.9.13 in hello-samza > > > Key: SAMZA-884 > URL: https://issues.apache.org/jira/browse/SAMZA-884 > Project: Samza > Issue Type: Bug > Components: hello-samza >Reporter: Navina Ramesh > Labels: newbie > Fix For: 0.10.1 > > Attachments: SAMZA-884-0.patch > > > As a result of jackson version upgrade in SAMZA-878 , the hello-samza job > breaks. We need to change the dependency there as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-882) Detect partition count changes in input streams
[ https://issues.apache.org/jira/browse/SAMZA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180226#comment-15180226 ] Yi Pan (Data Infrastructure) commented on SAMZA-882: [~jarradk], thanks for your input. IMO, the "dormant partition" problem you described would be better handled by a correct choice of hash function in partitioning. In your example, when you use user id as the partition key, there will be M number of users who are mapped to a single partition, based on some hash function. Hence, the "dormant partition" can only happen if *all* M users "leave the company". Usually, the probability for all user ids mapped to a single partition id are invalid is really low, with a good choice of hash function. > Detect partition count changes in input streams > --- > > Key: SAMZA-882 > URL: https://issues.apache.org/jira/browse/SAMZA-882 > Project: Samza > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Navina Ramesh >Assignee: Navina Ramesh > Fix For: 0.10.1 > > > This is a known issue where any change in the partition count in the upstream > affects the Samza job and it needs to be restarted. In such scenarios, we > experience data loss or incorrect processing because the application logic > depends on the partitioning strategy. It is worsened by the fact that we > don't even have a good mechanism to detect such a change. > As a first-step towards detection, I propose that we modify the stream > metadata cache maintained in Samza such that when there a change in partition > count, we increment a gauge metric. This way we can at least attach a hook to > monitor when this happens and take necessary actions. > However, in the long-term, we need to come up with a better strategy for > handling this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-813) Add Seek functionality to KeyValueStoreIterator
[ https://issues.apache.org/jira/browse/SAMZA-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176690#comment-15176690 ] Yi Pan (Data Infrastructure) commented on SAMZA-813: Hi, [~amitra123], thanks! I will try to get it reviewed by Fri this week. > Add Seek functionality to KeyValueStoreIterator > --- > > Key: SAMZA-813 > URL: https://issues.apache.org/jira/browse/SAMZA-813 > Project: Samza > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Amit Yadav >Assignee: Amit Yadav > Attachments: SAMZA-813.0.patch > > > The KeyValueIterator today does not support seeking to a particular key. In > some scenarios, seeking to a key provides huge benefits (E.g when there are > large number of deletes which causes issues- > https://www.facebook.com/groups/rocksdb.dev/permalink/604723469626171/) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-884) Upgrade Jackson to 1.9.13 in hello-samza
[ https://issues.apache.org/jira/browse/SAMZA-884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176663#comment-15176663 ] Yi Pan (Data Infrastructure) commented on SAMZA-884: Hm... sorry I missed that. This really reminds me of the following JIRA: SAMZA-205. > Upgrade Jackson to 1.9.13 in hello-samza > > > Key: SAMZA-884 > URL: https://issues.apache.org/jira/browse/SAMZA-884 > Project: Samza > Issue Type: Bug > Components: hello-samza >Reporter: Navina Ramesh > Labels: newbie > Fix For: 0.10.1 > > > As a result of jackson version upgrade in SAMZA-878 , the hello-samza job > breaks. We need to change the dependency there as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SAMZA-878) upgrade Jackson to 1.9.13
[ https://issues.apache.org/jira/browse/SAMZA-878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) resolved SAMZA-878. Resolution: Fixed Assignee: Yi Pan (Data Infrastructure) Fix Version/s: 0.10.1 > upgrade Jackson to 1.9.13 > - > > Key: SAMZA-878 > URL: https://issues.apache.org/jira/browse/SAMZA-878 > Project: Samza > Issue Type: Bug >Reporter: Shadi A. Noghabi >Assignee: Yi Pan (Data Infrastructure) > Fix For: 0.10.1 > > Attachments: jackson issue.PNG, rocksopts_avrohdfs.patch > > > Hadoop has upgraded their Jackson dependency to 1.9.13. Since we are > supporting Hadoop 1.6, we should upgrade our Jackson dependency to 1.9.13 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-878) upgrade Jackson to 1.9.13
[
https://issues.apache.org/jira/browse/SAMZA-878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172639#comment-15172639
]
Yi Pan (Data Infrastructure) commented on SAMZA-878:
Hi, guys, I apologize for the confusing due to the review comment
[rb43732|https://reviews.apache.org/r/43732]. This ticket is strictly stating
jackson 1.9.13, due to dependency report from
[here|http://hadoop.apache.org/docs/r2.6.1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/dependency-analysis.html].
To avoid confusion, I have applied the following patch:
{code}
diff --git a/gradle/dependency-versions.gradle
b/gradle/dependency-versions.gradle
index a34b072..47c71bf 100644
--- a/gradle/dependency-versions.gradle
+++ b/gradle/dependency-versions.gradle
@@ -20,7 +20,7 @@
elasticsearchVersion = "1.5.1"
jodaTimeVersion = "2.2"
joptSimpleVersion = "3.2"
- jacksonVersion = "1.8.5"
+ jacksonVersion = "1.9.13"
junitVersion = "4.8.1"
mockitoVersion = "1.8.4"
scalaTestVersion = "2.2.4"
{code}
[~edi_b...@yahoo.com], we can merge and update your patch based on this
version. Thanks!
> upgrade Jackson to 1.9.13
> -
>
> Key: SAMZA-878
> URL: https://issues.apache.org/jira/browse/SAMZA-878
> Project: Samza
> Issue Type: Bug
>Reporter: Shadi A. Noghabi
> Attachments: jackson issue.PNG, rocksopts_avrohdfs.patch
>
>
> Hadoop has upgraded their Jackson dependency to 1.9.13. Since we are
> supporting Hadoop 1.6, we should upgrade our Jackson dependency to 1.9.13
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SAMZA-880) Moving to github/pull-request for code review and check-in
Yi Pan (Data Infrastructure) created SAMZA-880: -- Summary: Moving to github/pull-request for code review and check-in Key: SAMZA-880 URL: https://issues.apache.org/jira/browse/SAMZA-880 Project: Samza Issue Type: Task Reporter: Yi Pan (Data Infrastructure) As per mailing list [discussion|http://mail-archives.apache.org/mod_mbox/samza-dev/201602.mbox/%3ccafvexu230gjja7z99lzn6bhnekypjrnof3kyc+dvgko_znq...@mail.gmail.com%3E], we are going to change the code review and commit process to use github pull request. This would require a few things: 1) incorporate kafka-merge-pr.py from [kafka repo|https://github.com/apache/kafka/blob/trunk/kafka-merge-pr.py] to Samza that include the merge steps for github pull requests to Apache Samza git repo 2) update the documentation in contributor corner for the new developer workflow -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-609) Properties not encoded properly
[
https://issues.apache.org/jira/browse/SAMZA-609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157367#comment-15157367
]
Yi Pan (Data Infrastructure) commented on SAMZA-609:
[~naveenatceg], is this fixed w/ the coordinator stream as well? It will only
be a problem if the config is passed via command line args, right?
> Properties not encoded properly
> ---
>
> Key: SAMZA-609
> URL: https://issues.apache.org/jira/browse/SAMZA-609
> Project: Samza
> Issue Type: Bug
> Components: container
>Affects Versions: 0.8.0
>Reporter: Naveen Somasundaram
>
> Passing a complex configuration:
> {noformat}
> filter.map.filter1.property=xpath("name") in ("uiBrowseStartup.ended",
> "subscription.ended", "uiStartup.ended") or xpath("category") =
> "uiIntent"
> {noformat}
>
> This property is understood by Java properties (Verified), however when used
> in Samza, it breaks with the following error:
> {noformat}
> Exception in thread "main" org.codehaus.jackson.JsonParseException:
> Unrecognized character escape ')' (code 41)
> at [Source: java.io.StringReader@409bf450; line: 1, column: 46]
> at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._handleUnrecognizedCharacterEscape(JsonParserMinimalBase.java:360)
> at
> org.codehaus.jackson.impl.ReaderBasedParser._decodeEscaped(ReaderBasedParser.java:1062)
> at
> org.codehaus.jackson.impl.ReaderBasedParser._finishString2(ReaderBasedParser.java:783)
> at
> org.codehaus.jackson.impl.ReaderBasedParser._finishString(ReaderBasedParser.java:760)
> at
> org.codehaus.jackson.impl.ReaderBasedParser.getText(ReaderBasedParser.java:85)
> at
> org.codehaus.jackson.map.deser.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:39)
> at
> org.codehaus.jackson.map.deser.MapDeserializer._readAndBind(MapDeserializer.java:235)
> at
> org.codehaus.jackson.map.deser.MapDeserializer.deserialize(MapDeserializer.java:165)
> at
> org.codehaus.jackson.map.deser.MapDeserializer.deserialize(MapDeserializer.java:25)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2395)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
> at
> org.apache.samza.config.serializers.JsonConfigSerializer$.fromJson(JsonConfigSerializer.scala:34)
> at
> org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:72)
> at org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
> {noformat}
> To reproduce, just paste the config in hello-samza as a property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (SAMZA-872) Remove unsafe log4j DailyRollingFileAppender from docs and hello-samza
[ https://issues.apache.org/jira/browse/SAMZA-872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) resolved SAMZA-872. Resolution: Fixed Fix Version/s: 0.10.1 > Remove unsafe log4j DailyRollingFileAppender from docs and hello-samza > -- > > Key: SAMZA-872 > URL: https://issues.apache.org/jira/browse/SAMZA-872 > Project: Samza > Issue Type: Bug >Reporter: Jake Maes >Assignee: Branislav Cogic > Labels: newbie > Fix For: 0.10.1 > > Attachments: SAMZA-872.0.patch, SAMZA-872.1.patch > > > DailyRollingFileAppender is known to have synchronization issues and cause > data loss. It even claims this in its own documentation > https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html > We shouldn't be referencing such a problematic appender in Samza. Instead, we > could reference RollingFileAppender with these settings: > > > > > > > > > Docs: > https://samza.apache.org/learn/documentation/0.10/jobs/logging.html > Hello Samza: > https://github.com/apache/samza-hello-samza/blob/master/src/main/resources/log4j.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-872) Remove unsafe log4j DailyRollingFileAppender from docs and hello-samza
[ https://issues.apache.org/jira/browse/SAMZA-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154604#comment-15154604 ] Yi Pan (Data Infrastructure) commented on SAMZA-872: +1. Merged and submitted both patches. Thanks! > Remove unsafe log4j DailyRollingFileAppender from docs and hello-samza > -- > > Key: SAMZA-872 > URL: https://issues.apache.org/jira/browse/SAMZA-872 > Project: Samza > Issue Type: Bug >Reporter: Jake Maes >Assignee: Branislav Cogic > Labels: newbie > Fix For: 0.10.1 > > Attachments: SAMZA-872.0.patch, SAMZA-872.1.patch > > > DailyRollingFileAppender is known to have synchronization issues and cause > data loss. It even claims this in its own documentation > https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html > We shouldn't be referencing such a problematic appender in Samza. Instead, we > could reference RollingFileAppender with these settings: > > > > > > > > > Docs: > https://samza.apache.org/learn/documentation/0.10/jobs/logging.html > Hello Samza: > https://github.com/apache/samza-hello-samza/blob/master/src/main/resources/log4j.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-864) allow job not to fail in case of incorrect number of partitions on checkpoint topic
[ https://issues.apache.org/jira/browse/SAMZA-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154574#comment-15154574 ] Yi Pan (Data Infrastructure) commented on SAMZA-864: [~boryas], thanks! I took the liberty to update the online doc adding a "CAUTION" note for this configuration. Since it is just doc change, I merged w/ your patch and committed. Thanks! > allow job not to fail in case of incorrect number of partitions on checkpoint > topic > --- > > Key: SAMZA-864 > URL: https://issues.apache.org/jira/browse/SAMZA-864 > Project: Samza > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Fix For: 0.10.1 > > Attachments: patch, rb43053-1.patch, rb43053.patch > > > Today, if there is an incorrect number of partitions on the job's checkpoint > topic, the job will fail right away. The topic should have 1 partition. > Unfortunately, there is no easy solution for this problem, since the number > of partitions on the topic cannot be decreased (or topic cannot be deleted). > We should add a config that will allow just to issue a warning in this case > and keep going. We also need to make sure the checkpoints are written to > partition 0 only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SAMZA-689) When Kafka has topic auto-creation turned on, checkpoint topics may be created w/ wrong partition numbers
[ https://issues.apache.org/jira/browse/SAMZA-689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) resolved SAMZA-689. Resolution: Won't Fix > When Kafka has topic auto-creation turned on, checkpoint topics may be > created w/ wrong partition numbers > - > > Key: SAMZA-689 > URL: https://issues.apache.org/jira/browse/SAMZA-689 > Project: Samza > Issue Type: Bug > Components: container >Affects Versions: 0.9.1 >Reporter: Yi Pan (Data Infrastructure) > > When Kafka topic auto-creation is turned on in the brokers, a fetch metadata > API call will result in topic auto-creation w/ default number of partitions. > In Samza code, we should always try to create the checkpoint/changelog topic > w/ the expected number of partitions first, then call fetch metadata to avoid > topic auto-creation creating the topic w/ un-wanted number of partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-689) When Kafka has topic auto-creation turned on, checkpoint topics may be created w/ wrong partition numbers
[ https://issues.apache.org/jira/browse/SAMZA-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154517#comment-15154517 ] Yi Pan (Data Infrastructure) commented on SAMZA-689: Hi, revisiting this one now and it seems to me that we can close as won't fix: # We have never been able to re-produce the issue w/ correct cluster configuration (i.e. no discrepancy between systems..consumer.zookeeper.connect vs systems..producer.bootstrap.servers) # With SAMZA-864, we have a configuration to allow the job working w/ checkpoint auto-created by mistake > When Kafka has topic auto-creation turned on, checkpoint topics may be > created w/ wrong partition numbers > - > > Key: SAMZA-689 > URL: https://issues.apache.org/jira/browse/SAMZA-689 > Project: Samza > Issue Type: Bug > Components: container >Affects Versions: 0.9.1 >Reporter: Yi Pan (Data Infrastructure) > > When Kafka topic auto-creation is turned on in the brokers, a fetch metadata > API call will result in topic auto-creation w/ default number of partitions. > In Samza code, we should always try to create the checkpoint/changelog topic > w/ the expected number of partitions first, then call fetch metadata to avoid > topic auto-creation creating the topic w/ un-wanted number of partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-876) Implemented AvroDataFileHdfsWriter and exposed several RocksDb config
[ https://issues.apache.org/jira/browse/SAMZA-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153079#comment-15153079 ] Yi Pan (Data Infrastructure) commented on SAMZA-876: Hi, [~edi_b...@yahoo.com], since you already provided RB for this ticket, please keep assignment to you to take the credit. :) Thanks! > Implemented AvroDataFileHdfsWriter and exposed several RocksDb config > - > > Key: SAMZA-876 > URL: https://issues.apache.org/jira/browse/SAMZA-876 > Project: Samza > Issue Type: Improvement > Components: container, kv-store >Affects Versions: 0.10.1 > Environment: RHEL 6.6, Java 1.8.0_72 >Reporter: Edi Bice >Assignee: Edi Bice >Priority: Minor > Labels: patch > Fix For: 0.10.1 > > Attachments: rocksopts_avrohdfs.patch > > > Implemented AvroDataFileHdfsWriter loosely fashioned after > BinarySequenceFileHDFSWriter. > Added ability to configure settings for RocksDB infoLogLevel, keepLogFileNum, > logFileTimeToRoll and maxLogFileSize. > Added ability to configure settings for RocksDB bloom filter, level > compactions etc based on RocksDb tuning guide -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-876) Implemented AvroDataFileHdfsWriter and exposed several RocksDb config
[ https://issues.apache.org/jira/browse/SAMZA-876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-876: --- Assignee: Edi Bice (was: Navina Ramesh) > Implemented AvroDataFileHdfsWriter and exposed several RocksDb config > - > > Key: SAMZA-876 > URL: https://issues.apache.org/jira/browse/SAMZA-876 > Project: Samza > Issue Type: Improvement > Components: container, kv-store >Affects Versions: 0.10.1 > Environment: RHEL 6.6, Java 1.8.0_72 >Reporter: Edi Bice >Assignee: Edi Bice >Priority: Minor > Labels: patch > Fix For: 0.10.1 > > Attachments: rocksopts_avrohdfs.patch > > > Implemented AvroDataFileHdfsWriter loosely fashioned after > BinarySequenceFileHDFSWriter. > Added ability to configure settings for RocksDB infoLogLevel, keepLogFileNum, > logFileTimeToRoll and maxLogFileSize. > Added ability to configure settings for RocksDB bloom filter, level > compactions etc based on RocksDb tuning guide -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-872) Remove unsafe log4j DailyRollingFileAppender from docs and hello-samza
[
https://issues.apache.org/jira/browse/SAMZA-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152903#comment-15152903
]
Yi Pan (Data Infrastructure) commented on SAMZA-872:
Thanks a lot [~banecogic]. Could you also post the patch for the documentation
change as well? The docs are in ${SAMZA_SRC_ROOT}/docs
Thanks!
> Remove unsafe log4j DailyRollingFileAppender from docs and hello-samza
> --
>
> Key: SAMZA-872
> URL: https://issues.apache.org/jira/browse/SAMZA-872
> Project: Samza
> Issue Type: Bug
>Reporter: Jake Maes
>Assignee: Branislav Cogic
> Labels: newbie
> Attachments: SAMZA-872.0.patch
>
>
> DailyRollingFileAppender is known to have synchronization issues and cause
> data loss. It even claims this in its own documentation
> https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> We shouldn't be referencing such a problematic appender in Samza. Instead, we
> could reference RollingFileAppender with these settings:
>
>
>
>
>
>
>
>
> Docs:
> https://samza.apache.org/learn/documentation/0.10/jobs/logging.html
> Hello Samza:
> https://github.com/apache/samza-hello-samza/blob/master/src/main/resources/log4j.xml
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SAMZA-873) Avoid unnecessary flushes in CachedStore
[
https://issues.apache.org/jira/browse/SAMZA-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152845#comment-15152845
]
Yi Pan (Data Infrastructure) commented on SAMZA-873:
+1. Merged and submitted. Thanks!
> Avoid unnecessary flushes in CachedStore
>
>
> Key: SAMZA-873
> URL: https://issues.apache.org/jira/browse/SAMZA-873
> Project: Samza
> Issue Type: Improvement
> Components: kv
>Affects Versions: 0.10.0
>Reporter: Nicolas Maquet
>Assignee: Nicolas Maquet
> Fix For: 0.10.1
>
> Attachments:
> 0001-SAMZA-873-Fix-CachedStore-to-not-call-flush-unnecess.patch
>
>
> The class {{org.apache.samza.storage.kv.CachedStore}} is currently calling
> {{store.flush()}} when evicting dirty entries. This in turn causes RocksDB to
> flush its memtables much more than necessary, causing slowdowns.
> In a mixed put / get workload, e.g. 2 gets for 1 put with an object cache
> size of 1000, RocksDB will flush its memtable roughly every 333 calls to
> put(); that is every time the eldest entry from the cache is dirty. In our
> benchmarks, this leads to a more than 20x drop in throughput.
> The attached patch fixes the issue as follows:
> - {{CachedStore.put()}} no longer flushes when evicting dirty entries.
> It calls {{store.putAll()}} with all dirty entries and resets the dirty list
> and count but does not call {{store.flush()}}.
> - Likewise, {{CachedStore.cache.removeEldestEntry()}} no longer flushes when
> evicting dirty entries.
> It calls {{store.putAll()}} on all dirty entries and resets the dirty list
> and count.
> - The behavior of {{CachedStore.flush()}} is unaffected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SAMZA-872) Remove unsafe log4j DailyRollingFileAppender from docs and hello-samza
[ https://issues.apache.org/jira/browse/SAMZA-872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-872: --- Assignee: Branislav Cogic > Remove unsafe log4j DailyRollingFileAppender from docs and hello-samza > -- > > Key: SAMZA-872 > URL: https://issues.apache.org/jira/browse/SAMZA-872 > Project: Samza > Issue Type: Bug >Reporter: Jake Maes >Assignee: Branislav Cogic > Labels: newbie > Attachments: SAMZA-872.0.patch > > > DailyRollingFileAppender is known to have synchronization issues and cause > data loss. It even claims this in its own documentation > https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html > We shouldn't be referencing such a problematic appender in Samza. Instead, we > could reference RollingFileAppender with these settings: > > > > > > > > > Docs: > https://samza.apache.org/learn/documentation/0.10/jobs/logging.html > Hello Samza: > https://github.com/apache/samza-hello-samza/blob/master/src/main/resources/log4j.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (SAMZA-776) building instructions have bad URL - http instead of https
[ https://issues.apache.org/jira/browse/SAMZA-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) reassigned SAMZA-776: -- Assignee: Yi Pan (Data Infrastructure) (was: Edi Bice) > building instructions have bad URL - http instead of https > -- > > Key: SAMZA-776 > URL: https://issues.apache.org/jira/browse/SAMZA-776 > Project: Samza > Issue Type: Bug > Components: docs >Affects Versions: 0.10.0 >Reporter: Edi Bice >Assignee: Yi Pan (Data Infrastructure) >Priority: Trivial > Attachments: SAMZA-776.0.patch > > Original Estimate: 10m > Remaining Estimate: 10m > > Last section of this page http://samza.apache.org/startup/download/ instructs > to download the code via > git clone http://git-wip-us.apache.org/repos/asf/samza.git > That fails with RPC error. The correct URL and command to use is > git clone https://git-wip-us.apache.org/repos/asf/samza.git -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SAMZA-776) building instructions have bad URL - http instead of https
[ https://issues.apache.org/jira/browse/SAMZA-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Pan (Data Infrastructure) updated SAMZA-776: --- Assignee: Aleksandar Bircakovic (was: Yi Pan (Data Infrastructure)) > building instructions have bad URL - http instead of https > -- > > Key: SAMZA-776 > URL: https://issues.apache.org/jira/browse/SAMZA-776 > Project: Samza > Issue Type: Bug > Components: docs >Affects Versions: 0.10.0 >Reporter: Edi Bice >Assignee: Aleksandar Bircakovic >Priority: Trivial > Attachments: SAMZA-776.0.patch > > Original Estimate: 10m > Remaining Estimate: 10m > > Last section of this page http://samza.apache.org/startup/download/ instructs > to download the code via > git clone http://git-wip-us.apache.org/repos/asf/samza.git > That fails with RPC error. The correct URL and command to use is > git clone https://git-wip-us.apache.org/repos/asf/samza.git -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-873) Avoid unnecessary flushes in CachedStore
[
https://issues.apache.org/jira/browse/SAMZA-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152725#comment-15152725
]
Yi Pan (Data Infrastructure) commented on SAMZA-873:
Cool! I will take a look asap.
Thanks!
> Avoid unnecessary flushes in CachedStore
>
>
> Key: SAMZA-873
> URL: https://issues.apache.org/jira/browse/SAMZA-873
> Project: Samza
> Issue Type: Improvement
> Components: kv
>Affects Versions: 0.10.0
>Reporter: Nicolas Maquet
>Assignee: Nicolas Maquet
> Fix For: 0.10.1
>
> Attachments:
> 0001-SAMZA-873-Fix-CachedStore-to-not-call-flush-unnecess.patch
>
>
> The class {{org.apache.samza.storage.kv.CachedStore}} is currently calling
> {{store.flush()}} when evicting dirty entries. This in turn causes RocksDB to
> flush its memtables much more than necessary, causing slowdowns.
> In a mixed put / get workload, e.g. 2 gets for 1 put with an object cache
> size of 1000, RocksDB will flush its memtable roughly every 333 calls to
> put(); that is every time the eldest entry from the cache is dirty. In our
> benchmarks, this leads to a more than 20x drop in throughput.
> The attached patch fixes the issue as follows:
> - {{CachedStore.put()}} no longer flushes when evicting dirty entries.
> It calls {{store.putAll()}} with all dirty entries and resets the dirty list
> and count but does not call {{store.flush()}}.
> - Likewise, {{CachedStore.cache.removeEldestEntry()}} no longer flushes when
> evicting dirty entries.
> It calls {{store.putAll()}} on all dirty entries and resets the dirty list
> and count.
> - The behavior of {{CachedStore.flush()}} is unaffected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SAMZA-873) Avoid unnecessary flushes in CachedStore
[
https://issues.apache.org/jira/browse/SAMZA-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Pan (Data Infrastructure) updated SAMZA-873:
---
Assignee: Nicolas Maquet
> Avoid unnecessary flushes in CachedStore
>
>
> Key: SAMZA-873
> URL: https://issues.apache.org/jira/browse/SAMZA-873
> Project: Samza
> Issue Type: Improvement
> Components: kv
>Affects Versions: 0.10.0
>Reporter: Nicolas Maquet
>Assignee: Nicolas Maquet
> Fix For: 0.10.1
>
> Attachments:
> 0001-SAMZA-873-Fix-CachedStore-to-not-call-flush-unnecess.patch
>
>
> The class {{org.apache.samza.storage.kv.CachedStore}} is currently calling
> {{store.flush()}} when evicting dirty entries. This in turn causes RocksDB to
> flush its memtables much more than necessary, causing slowdowns.
> In a mixed put / get workload, e.g. 2 gets for 1 put with an object cache
> size of 1000, RocksDB will flush its memtable roughly every 333 calls to
> put(); that is every time the eldest entry from the cache is dirty. In our
> benchmarks, this leads to a more than 20x drop in throughput.
> The attached patch fixes the issue as follows:
> - {{CachedStore.put()}} no longer flushes when evicting dirty entries.
> It calls {{store.putAll()}} with all dirty entries and resets the dirty list
> and count but does not call {{store.flush()}}.
> - Likewise, {{CachedStore.cache.removeEldestEntry()}} no longer flushes when
> evicting dirty entries.
> It calls {{store.putAll()}} on all dirty entries and resets the dirty list
> and count.
> - The behavior of {{CachedStore.flush()}} is unaffected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SAMZA-873) Avoid unnecessary flushes in CachedStore
[
https://issues.apache.org/jira/browse/SAMZA-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152687#comment-15152687
]
Yi Pan (Data Infrastructure) commented on SAMZA-873:
[~nico...@movio.co], could you create a reviewboard request for the patch? It
would be easier for review.
Thanks!
> Avoid unnecessary flushes in CachedStore
>
>
> Key: SAMZA-873
> URL: https://issues.apache.org/jira/browse/SAMZA-873
> Project: Samza
> Issue Type: Improvement
> Components: kv
>Affects Versions: 0.10.0
>Reporter: Nicolas Maquet
> Attachments:
> 0001-SAMZA-873-Fix-CachedStore-to-not-call-flush-unnecess.patch
>
>
> The class {{org.apache.samza.storage.kv.CachedStore}} is currently calling
> {{store.flush()}} when evicting dirty entries. This in turn causes RocksDB to
> flush its memtables much more than necessary, causing slowdowns.
> In a mixed put / get workload, e.g. 2 gets for 1 put with an object cache
> size of 1000, RocksDB will flush its memtable roughly every 333 calls to
> put(); that is every time the eldest entry from the cache is dirty. In our
> benchmarks, this leads to a more than 20x drop in throughput.
> The attached patch fixes the issue as follows:
> - {{CachedStore.put()}} no longer flushes when evicting dirty entries.
> It calls {{store.putAll()}} with all dirty entries and resets the dirty list
> and count but does not call {{store.flush()}}.
> - Likewise, {{CachedStore.cache.removeEldestEntry()}} no longer flushes when
> evicting dirty entries.
> It calls {{store.putAll()}} on all dirty entries and resets the dirty list
> and count.
> - The behavior of {{CachedStore.flush()}} is unaffected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SAMZA-864) allow job not to fail in case of incorrect number of partitions on checkpoint topic
[ https://issues.apache.org/jira/browse/SAMZA-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149969#comment-15149969 ] Yi Pan (Data Infrastructure) commented on SAMZA-864: Also, I noticed that you have added one configure variable to the job configuration. We should update the document w/ the newly added configure variable as well. > allow job not to fail in case of incorrect number of partitions on checkpoint > topic > --- > > Key: SAMZA-864 > URL: https://issues.apache.org/jira/browse/SAMZA-864 > Project: Samza > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: patch, rb43053.patch > > > Today, if there is an incorrect number of partitions on the job's checkpoint > topic, the job will fail right away. The topic should have 1 partition. > Unfortunately, there is no easy solution for this problem, since the number > of partitions on the topic cannot be decreased (or topic cannot be deleted). > We should add a config that will allow just to issue a warning in this case > and keep going. We also need to make sure the checkpoints are written to > partition 0 only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMZA-864) allow job not to fail in case of incorrect number of partitions on checkpoint topic
[
https://issues.apache.org/jira/browse/SAMZA-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149942#comment-15149942
]
Yi Pan (Data Infrastructure) commented on SAMZA-864:
Thanks, [~boryas]. I applied the patch and it has the following unit test
failure:
{code}
:samza-kafka_2.10:test
testFailOnTopicValidation FAILED
java.lang.AssertionError: Expected a KafkaUtilException for invalid number
of partitions in the topic.
at org.junit.Assert.fail(Assert.java:91)
at
org.apache.samza.checkpoint.kafka.TestKafkaCheckpointManager.testFailOnTopicValidation(TestKafkaCheckpointManager.scala:216)
{code}
Could you check again?
Thanks!
> allow job not to fail in case of incorrect number of partitions on checkpoint
> topic
> ---
>
> Key: SAMZA-864
> URL: https://issues.apache.org/jira/browse/SAMZA-864
> Project: Samza
> Issue Type: Bug
>Reporter: Boris Shkolnik
>Assignee: Boris Shkolnik
> Attachments: patch, rb43053.patch
>
>
> Today, if there is an incorrect number of partitions on the job's checkpoint
> topic, the job will fail right away. The topic should have 1 partition.
> Unfortunately, there is no easy solution for this problem, since the number
> of partitions on the topic cannot be decreased (or topic cannot be deleted).
> We should add a config that will allow just to issue a warning in this case
> and keep going. We also need to make sure the checkpoints are written to
> partition 0 only.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SAMZA-836) 0.10.0 :samza-kv-rocksdb_2.10:test fails
[
https://issues.apache.org/jira/browse/SAMZA-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15145609#comment-15145609
]
Yi Pan (Data Infrastructure) commented on SAMZA-836:
Finally got it to work! The problem is identified as the following:
When running the unit test in Linux, w/ JDK8, somehow, invoking the following
JNI API does not invoke NativeLibraryLoader in rocksdbjni-3.13.1:
{code}
val flushOptions = new FlushOptions().setWaitForFlush(true)
{code}
while calling {code}new Options(){code} actually invoke NativeLibraryLoader
Hence, I moved the creation of FlushOptions object after creating Options
object and the problem goes away!
It is a weird problem and I will report to RocksDb community.
Patch attached.
> 0.10.0 :samza-kv-rocksdb_2.10:test fails
>
>
> Key: SAMZA-836
> URL: https://issues.apache.org/jira/browse/SAMZA-836
> Project: Samza
> Issue Type: Bug
>Affects Versions: 0.10.0
> Environment: Linux
>Reporter: Boris Shkolnik
>Assignee: Yi Pan (Data Infrastructure)
> Attachments: SAMZA-836-0.patch
>
>
> First time I ran the ./gradlew clean check one test failed:
> Gradle test > org.apache.samza.storage.kv.TestRocksDbKeyValueStore.testFlush
> > no output captured
> testFlush FAILED
> java.lang.UnsatisfiedLinkError:
> org.rocksdb.FlushOptions.newFlushOptions()V
> at org.rocksdb.FlushOptions.newFlushOptions(Native Method)
> at org.rocksdb.FlushOptions.(FlushOptions.java:14)
> at
> org.apache.samza.storage.kv.TestRocksDbKeyValueStore.testFlush(TestRocksDbKeyValueStore.scala:73)
> :samza-kv-rocksdb_2.10 -- There are test failures!
> I noticed that I have some mvn cache with older version of rocksdjni. I
> deleted it and seems to pass now. I am still not 100% sure that was the
> reason. Filing the bug for tracking purposes.
> ~/.m2/repository/org/rocksdb
> ~/.m2/repository/org/rocksdb/rocksdbjni
> ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.jar
> ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.jar.sha1
> ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.pom
> ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.pom.sha1
> ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.pom.lastUpdated
> ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.jar.lastUpdated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SAMZA-836) 0.10.0 :samza-kv-rocksdb_2.10:test fails
[ https://issues.apache.org/jira/browse/SAMZA-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15145658#comment-15145658 ] Yi Pan (Data Infrastructure) commented on SAMZA-836: The issue is reported to RocksDb github as [issue 989| https://github.com/facebook/rocksdb/issues/989] > 0.10.0 :samza-kv-rocksdb_2.10:test fails > > > Key: SAMZA-836 > URL: https://issues.apache.org/jira/browse/SAMZA-836 > Project: Samza > Issue Type: Bug >Affects Versions: 0.10.0 > Environment: Linux >Reporter: Boris Shkolnik >Assignee: Yi Pan (Data Infrastructure) > Attachments: SAMZA-836-0.patch > > > First time I ran the ./gradlew clean check one test failed: > Gradle test > org.apache.samza.storage.kv.TestRocksDbKeyValueStore.testFlush > > no output captured > testFlush FAILED > java.lang.UnsatisfiedLinkError: > org.rocksdb.FlushOptions.newFlushOptions()V > at org.rocksdb.FlushOptions.newFlushOptions(Native Method) > at org.rocksdb.FlushOptions.(FlushOptions.java:14) > at > org.apache.samza.storage.kv.TestRocksDbKeyValueStore.testFlush(TestRocksDbKeyValueStore.scala:73) > :samza-kv-rocksdb_2.10 -- There are test failures! > I noticed that I have some mvn cache with older version of rocksdjni. I > deleted it and seems to pass now. I am still not 100% sure that was the > reason. Filing the bug for tracking purposes. > ~/.m2/repository/org/rocksdb > ~/.m2/repository/org/rocksdb/rocksdbjni > ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.jar > ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.jar.sha1 > ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.pom > ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.pom.sha1 > ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.pom.lastUpdated > ~/.m2/repository/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.jar.lastUpdated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
