GitHub user voronenko-da opened a pull request:
https://github.com/apache/samza/pull/631
Invalid partition calculation in KafkaSystemProducer 0.14.1
If you use a byte array as the key, the key.hashCode() will be different
all the time. Proposal to remove the preliminary calculation of the partition.
https://github.com/apache/samza/blob/master/samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala
val partitionKey = if (envelope.getPartitionKey != null)
KafkaUtil.getIntegerPartitionKey(envelope, partitions) else null
https://github.com/apache/samza/blob/master/samza-kafka/src/main/scala/org/apache/samza/util/KafkaUtil.scala
def getIntegerPartitionKey(envelope: OutgoingMessageEnvelope, partitions:
java.util.List[PartitionInfo]): Integer = {
val numPartitions = partitions.size
abs(envelope.getPartitionKey.hashCode()) % numPartitions
}
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/samza 0.14.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/631.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #631
----
commit 3f2c0775bfc69e7464f45343503c5e51310d55dd
Author: xiliu <xiliu@...>
Date: 2018-05-07T22:25:35Z
Trigger notification to github of 0.14.1 branch
commit 5ca7fac5767496a5c22ed6cc3096f366c1e304a9
Author: xiliu <xiliu@...>
Date: 2018-05-07T23:11:59Z
SAMZA-1702: Prepare 0.14.1 release on the 0.14.1 branch
Author: xiliu <[email protected]>
Reviewers: Prateek M <[email protected]>
Closes #505 from xinyuiscool/SAMZA-1702
commit 06039b789b8134c4dbbe7883b34722427b3f46c8
Author: xiliu <xiliu@...>
Date: 2018-05-08T00:06:35Z
SAMZA-1703: Disable flaky test
TestEmbeddedTaggedRateLimiter.testAcquireWithTimeout
Author: xiliu <[email protected]>
Reviewers: Boris S <[email protected]>
Closes #507 from xinyuiscool/SAMZA-1703
(cherry picked from commit 9ac221e4e99d3e576c7fad03d0b83eeb489564e9)
Signed-off-by: xiliu <[email protected]>
commit 4b19efefec78ec6d2f7352dfa1972a9d04181861
Author: xiliu <xiliu@...>
Date: 2018-05-08T01:14:29Z
SAMZA-1704: Fix compatibility issues with scala 2.12
Need to add override keyword for overriding a method in scala 2.12.
Author: xiliu <[email protected]>
Reviewers: Prateek M <[email protected]>
Closes #508 from xinyuiscool/SAMZA-1704
(cherry picked from commit 31a9e4aea8a8233853334b7ed70da8694f42bd52)
Signed-off-by: xiliu <[email protected]>
commit 44d6f40f7d8d331e40752fc369a83fc69edf7867
Author: xinyuiscool <xiliu@...>
Date: 2018-05-09T00:48:55Z
SAMZA-1705: Switch to use snapshot in iterable impl of RocksDb
We should use rocksDb.snapshot() method to keep the snapshot and creates a
new iterator with it all the time. The perf shows a little bit more expensive
but mostly on par with range iterator query.
Author: xinyuiscool <[email protected]>
Reviewers: Jagadish V <[email protected]>
Closes #510 from xinyuiscool/SAMZA-1705
(cherry picked from commit 89beb1fccb01c781a4de905d57a4bd99df25577a)
Signed-off-by: xiliu <[email protected]>
commit 64ea33d7d89c5f7f934a0cc8290f3ad3f925e98a
Author: Hai Lu <halu@...>
Date: 2018-05-09T22:32:44Z
SAMZA-1706: lazy initialization for eventhub system producer
We are seeing slow shutdown issue for eventhub system producers for users
who only use eventhub consumer (but then Samza system creates both consumer and
producer together no matter what). As a workaround, add lazy initialization to
the producer to avoid the slow shutdown
Author: Hai Lu <[email protected]>
Reviewers: Jagadish <[email protected]>
Closes #511 from lhaiesp/master
commit adb514aaee046bf760e4f992c45e1f4582bb759d
Author: Prateek Maheshwari <pmaheshw@...>
Date: 2018-05-10T16:21:12Z
Fixed test failure for TestRocksDbKeyValueStoreJava#testIterate
RocksDB Snapshots and any iterators obtained from them need to be closed
before the store is closed.
Otherwise the process aborts with the following message (at least on OSX):
`Assertion failed: (is_last_reference), function ~ColumnFamilyData, file
db/column_family.cc, line 457.`
Author: Prateek Maheshwari <[email protected]>
Reviewers: Jagadish V <[email protected]>
Closes #513 from prateekm/rocksdb-test-fi
commit 1e1330cefbcfc7f2ff8e20c8ff19c465f7df4b66
Author: Prateek Maheshwari <pmaheshw@...>
Date: 2018-05-10T19:16:33Z
Fixed test failure for TestRocksDbKeyValueStoreJava#testPerf
Iterators (incl. those obtained from snapshots) must be closed before store
close.
Author: Prateek Maheshwari <[email protected]>
Reviewers: Xinyu Liu <[email protected]>
Closes #514 from prateekm/rocksdb-test-fi
(cherry picked from commit 54c690e6fc52ac13d1c59fe1bd1c34f35f710164)
Signed-off-by: xiliu <[email protected]>
commit b2ca67ea19becac1ea8c7a73d544438e75f94784
Author: xinyuiscool <xiliu@...>
Date: 2018-05-10T21:19:08Z
Remove the iterable interface from KeyValueSnapshot
The iterable interface makes it hard for the users to close it after using.
Author: xinyuiscool <[email protected]>
Reviewers: Prateek M <[email protected]>
Closes #516 from xinyuiscool/kv-snapshot
(cherry picked from commit e204d298e84f549ecf4d8d9d642304502f046c29)
Signed-off-by: xiliu <[email protected]>
commit 973e0735e076101fa9c05e88df999cfdd289fd0a
Author: Yi Pan (Data Infrastructure) <yipan@...>
Date: 2018-05-17T05:19:37Z
SAMZA-1715: Unit test for Kafka admin deletedMessagesCalled() fail
Test locally and works.
Author: Yi Pan (Data Infrastructure) <[email protected]>
Reviewers: Jagadish <[email protected]>
Closes #523 from nickpan47/fix-unittest-deleted-messages
commit 69e63d8153b89cf47c618b3f4e871609d7b18a02
Author: Shanthoosh Venkataraman <santhoshvenkat1988@...>
Date: 2018-05-18T19:29:24Z
SAMZA-1720: Remove javafx.util dependency from samza-sql tests.
In samza-sql module, currently few test
classes(`TestSamzaSqlRelMessageSerde` and `TestSamzaSqlRelRecordSerde`) are
dependent upon `javafx.util.Pair` class(coming from `javafx` module).
`javafx.util.Pair` is not supported by default in all JDK builds(example;
open-jdk java-8 doesn't support `javafx` module) and it belongs to `javafx`
package which is primarily used for developing GUI applications. This
dependency is removed and replaced with `Pair` class from `apache-commons`.
Author: Shanthoosh Venkataraman <[email protected]>
Reviewers: Jagadish V <[email protected]>
Closes #527 from shanthoosh/SAMZA-1720
(cherry picked from commit 3b9e14be3aa5fc2b6003b7e244fba466db34236f)
Signed-off-by: xiliu <[email protected]>
commit 9bc03f7ab32199a15af157cc515d897917694cb5
Author: Srinivasulu Punuru <spunuru@...>
Date: 2018-05-18T19:31:44Z
Implementing the fetchSinkInfo in ConfigBasedIOResolver
1. I think we missed implementing the fetchSinkInfo method in the
ConfigBasedResolver when the API was introduced which is breaking the samza sql
console tool. This fixes it.
2. latest release of mac removed realpath so the command line tools are
broken. Removed the usage of realpath to fix these tools.
Thanks to nickpan47 for identifying these problems.
Author: Srinivasulu Punuru <[email protected]>
Reviewers: Yi Pan <[email protected]>
Closes #528 from srinipunuru/release-fix.1
(cherry picked from commit 171793b69b33081fc6277c9505b3055f79fcb4b7)
Signed-off-by: xiliu <[email protected]>
commit a652e6f5230a297f3d991c6441311ef55e562d9a
Author: xiliu <xiliu@...>
Date: 2018-05-25T22:18:38Z
Remove git checkout latest from the docs in 0.14.1 branch
----
---
