----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35445/ -----------------------------------------------------------
(Updated July 26, 2015, 10:31 p.m.) Review request for samza. Repository: samza Description ------- SAMZA-693: Very basic HDFS Producer service for Samza Diffs (updated) ----- build.gradle 0852adc samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala PRE-CREATION samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemAdmin.scala PRE-CREATION samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemFactory.scala PRE-CREATION samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducer.scala PRE-CREATION samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducerMetrics.scala PRE-CREATION samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/BinarySequenceFileHdfsWriter.scala PRE-CREATION samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/Bucketer.scala PRE-CREATION samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/HdfsWriter.scala PRE-CREATION samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/JobNameDateTimeBucketer.scala PRE-CREATION samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/SequenceFileHdfsWriter.scala PRE-CREATION samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/TextSequenceFileHdfsWriter.scala PRE-CREATION samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-text.properties PRE-CREATION samza-hdfs/src/test/resources/samza-hdfs-test-batch-job.properties PRE-CREATION samza-hdfs/src/test/resources/samza-hdfs-test-job-text.properties PRE-CREATION samza-hdfs/src/test/resources/samza-hdfs-test-job.properties PRE-CREATION samza-hdfs/src/test/scala/org/apache/samza/system/hdfs/TestHdfsSystemProducerTestSuite.scala PRE-CREATION settings.gradle 19bff97 Diff: https://reviews.apache.org/r/35445/diff/ Testing ------- New unit test, but it's fairly rudimentary. Passes "./gradlew test" and "./gradlew check" This only supplies an HDFS Producer, and this producer only writes SequenceFiles of ByteWriteables so far. If the patch were accepted as-is, I'd suggest future tickets for a matching HDFS Consumer, and a pluggable set of output formats, configurable via HdfsConfig settings. On the upside, this patch has been tested on a real cluster with real data, using several serdes, with good results. Thanks, Eli Reisman
