----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43732/ -----------------------------------------------------------
Review request for samza. Repository: samza Description ------- https://issues.apache.org/jira/browse/SAMZA-876 Implemented AvroDataFileHdfsWriter fashioned loosely after BinarySequenceFileHDFSWriter. Exposed several RocksDb configuration options (recommended in RocksDb tuning guide): rocksdb.log.level rocksdb.log.keepfilenum rocksdb.log.timetoroll rocksdb.log.maxfilesize rocksdb.bloomfilter.bits rocksdb.max.background.compactions rocksdb.max.background.flushes rocksdb.num.write.buffers rocksdb.target.file.size.base rocksdb.max.bytes.level.base Diffs ----- gradle.properties 16e1f5d gradle/dependency-versions.gradle 52e25aa samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala 7993119 samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/AvroDataFileHdfsWriter.scala PRE-CREATION samza-kv-rocksdb/src/main/java/org/apache/samza/storage/kv/RocksDbOptionsHelper.java d4f765c Diff: https://reviews.apache.org/r/43732/diff/ Testing ------- I am using AvroDataFileHdfsWriter at the end of my pipeline. I feed the generated avro files to Apache Samoa. Have processed millions of records successfully. The RocksDb config changes are older and were used and verified to be working when originally implemented. Thanks, Edi Bice
