Also, the PUTs are taking 10x of GETs is what baffles me a little bit. We’re
running on SSDs and here is our config:
stores.stage- store.write.batch.size=100000
stores.stage- store.object.cache.size=200000
stores.stage- store.rocksdb.num.write.buffers=3
stores.stage- store.rocksdb.compaction.style=level
stores.stage- store.rocksdb.block.size.bytes=16384
stores.stage-auctions-store.container.write.buffer.size.bytes=268435456 #256MB
stores.stage-auctions-store.container.cache.size.bytes=268435456 #256MB
This is for a container that has 1 task per container since our earlier 2
tasks/container experiment did not look too promising. Each message is ~ 1KB.
We do not have a lot of cache hits (cache hit % ~ 10% or less) and most GETs do
end up going to the KeyValueStorageEngine but even with that get-ns is ~
25,000ns but put-ns can be as high as 200,000ns
I can send other metrics, if they are useful.
On 3/9/17, 11:45 AM, "Ankit Malhotra" <amalho...@appnexus.com> wrote:
Replies inline
On 3/9/17, 11:24 AM, "Jagadish Venkatraman" <jagadish1...@gmail.com> wrote:
I understand you are receiving messages from *all* partitions (but fewer
messages from some partitions).
Some questions:
1. Is it possible that you may have saturated the capacity of the entire
container?
>> Possible but what metric/metrics would tell me that?
2. What is the time you spend inside *process* and *window* for the
affected container? (How does it compare with other containers?). The
metrics of interest are *process_ns* and *window_ns.*
>> process-ns is ~ 200,000ns window-ns happens once every 60 secs.and is <
1 ms
3. What is the number of messages per-second you process for the
affected
containers ? (How does it compare with other containers?). The metric of
interest is *process-calls*. You can also look at per-task process calls
<https://github.com/apache/samza/blob/master/samza-core/src/main/scala/org/apache/samza/container/TaskInstanceMetrics.scala>,
and per-partition messages read
<https://github.com/apache/samza/blob/master/samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemConsumerMetrics.scala>
>> I was quoting process-calls from the metrics registry. For task 2, its
lesser than those for task 1 as I stated earlier.
.
4. How many partitions do you have on the input topics?
>> 100 each that we are trying to do in 50 containers (2 tasks each, with a
pool of size 5)
4.1 Have you tried increasing the # of partitions, and re-deploying with
more containers? ( I take it that you are running on Yarn).
>> No, since we don’t have the ability to change input partitions right now.
*>> Another thing to add is our amount of time being blocked is
generally
quite high which we believe is mainly because our processing is not fast
enough? *
5. What metric are you referring to here? Do you mean the time spent in
the
*process* and *window* call? If you're not processing "fast enough", it
will be helpful to instrument where you are spending most of the time.
>> Referring to block-ns. From my understanding, since the stores are
synchronized, once the run loop asks the tasks to process, it then waits
(blocks) until at least one of them is ready to receive more. So multiple
threads may not really be helping us?
You can rely on Samza's MetricRegistry to configure / report your custom
metrics.
>> Most of the time spent is in the get and put from RocksDB. According to
the samza metrics (KeyValueStorageEngine.store-put-ns), our PUTs are much more
expensive than GETs. Any idea on how one would go about profiling that? We are
using Level compaction with 3 buffers each 500M. Average input message is ~ 1KB
and we commit (thus, flush the store) every 90 seconds. We are also using an
object cache of 2million objects with a write cache of 1m objects.
Input messages are ~ 5k/sec/partition aggregate for both input streams that
we are joining.
On Thu, Mar 9, 2017 at 5:20 AM, Ankit Malhotra <amalho...@appnexus.com>
wrote:
> Replies inline.
>
> --
> Ankit
>
> > On Mar 9, 2017, at 12:34 AM, Jagadish Venkatraman <
> jagadish1...@gmail.com> wrote:
> >
> > We can certainly help you debug this more. Some questions:
> >
> > 1. Are you processing messages (at all) from the "suffering"
containers?
> > (You can verify that by observing metrics/ logging etc.)
> Processing messages for sure. But mainly from one of the 2 partitions
that
> the container is reading from.
>
> The overall number of process calls for task 1 is much higher than the
> process calls for task 2 and the diff is approx the lag on the
container
> which is all from task 2.
> >
> > 2. If you are indeed processing messages, is it possible the
impacted
> > containers not able to keep up with the surge in the data? You can
try
> > re-partitioning your input topics (and increasing the number of
> containers)
> We were trying the async loop by having 2 tasks be on the same
container
> and multiple threads processing messages. The process is a simple
inner
> join with a get and put into the RocksDB store.
>
> We saw that both get and put for the suffering task was higher than
the
> task that was chugging log.
> >
> > 3. If you are not processing messages, maybe can you provide us
with your
> > stack trace? It will be super-helpful to find out if (or where)
> containers
> > are stuck.
> >
>
> Another thing to add is our amount of time being blocked is generally
> quite high which we believe is mainly because our processing is not
fast
> enough? To add more color, our rocks store's average get across all
tasks
> is around 20,000ns BUT average put is 5X or more. We have the object
cache
> enabled.
>
> > Thanks,
> > Jagadish
> >
> >
> > On Wed, Mar 8, 2017 at 1:05 PM, Ankit Malhotra
<amalho...@appnexus.com>
> > wrote:
> >
> >> Hi,
> >>
> >> While joining streams from 2 partitions to join 2 streams, we see
that
> >> some containers start suffering in that, lag (messages behind high
> >> watermark) for one of the tasks starts sky rocketing while the
other
> one is
> >> ~ 0.
> >>
> >> We are using default values for buffer sizes, fetch threshold, are
> using 4
> >> threads as part of the pool and are using the default
> >> RoundRobinMessageChooser.
> >>
> >> Happy to share more details/config if it can help to debug this
further.
> >>
> >> Thanks
> >> Ankit
> >>
> >>
> >>
> >>
> >>
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
