Re: Review Request 41365: SAMZA-838: negative rocksdb.ttl.ms is not handled correctly

2015-12-15 Thread Navina Ramesh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41365/#review110553
---

Ship it!


One nit. Otherwise, looks good!


samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala
 (line 54)


I don't think mentioning RocksDB wiki is necessary. You can change it to:  
"Non-positive TTL for RocksDB implies infinite TTL for the data."


- Navina Ramesh


On Dec. 14, 2015, 9:04 p.m., Tao Feng wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41365/
> ---
> 
> (Updated Dec. 14, 2015, 9:04 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> rocskdb ttl is handled per RocksDB 
> semantics(https://github.com/facebook/rocksdb/wiki/Time-to-Live) which zero 
> or negative ttl is same as infinity ttl.
> 
> 
> Diffs
> -
> 
>   
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala
>  211fc3be1e168f1f92812406785b39b5a3fd9174 
> 
> Diff: https://reviews.apache.org/r/41365/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew clean build &&  ./gradlew checkstyleMain checkstyleTest
> 
> 
> Thanks,
> 
> Tao Feng
> 
>

Re: Random connection errors

2015-12-15 Thread Kishore N C 
Hi Yi Pan,

I'm using Samza 0.9.1 and Kafka 0.8.2.1. Here's an example of a full task
log:

https://gist.githubusercontent.com/kishorenc/5d65f114a50b9ef6a6b3/raw/5b9ecffdd1af831f713e8b41e5b77e5b881e8173/gistfile1.txt

You will find "java.nio.channels.ClosedChannelException" towards the end.

One additional thing to mention here is that I detect when the Samza job
has completed (by checking offsets periodically in the window() calls) and
issue a "taskCoordinator.shutdown(RequestScope.CURRENT_TASK)" in my code.

In some other cases, an error like:

"2015-12-15 10:09:54 ClientUtils$ [WARN] Fetching topic metadata with
correlation id 10 for topics [Set(TOPIC_TWO)] from broker
[id:2,host:10.181.18.171,port:9082] failed
java.nio.channels.ClosedChannelException"

occurs right at the start of the job and the task would just refuse to
start. Unfortunately, I don't have the logs for such a container anymore.

Thanks,

Kishore.

On Tue, Dec 15, 2015 at 12:31 AM, Yi Pan  wrote:

> Hi, Kishore,
>
> First, I would like to ask which version of Samza you are running? And if
> you can attach the log and config of your container (i.e. I assume the log
> you attached here is a container log?), it would be greatly helpful.
>
> Thanks a lot!
>
> -Yi
>
> On Mon, Dec 14, 2015 at 5:07 AM, Kishore N C  wrote:
>
> > Hi,
> >
> > I have a 25 node Samza cluster and I am running a job on a dataset of a
> > billion records that is backed by a 7 node Kafka cluster.
> >
> > Some of the tasks on some of the Samza nodes don't seem to start at all
> > (while other tasks run fine on other nodes). The specific error message I
> > see is in the task log is:
> >
> > 2015-12-14 12:50:50 ClientUtils$ [INFO] Fetching metadata from broker
> > id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s)
> > Set(TOPIC_ONE, TOPIC_TWO)
> > 2015-12-14 12:50:50 SyncProducer [INFO] Connected to 10.181.18.87:9082
> for
> > producing
> > 2015-12-14 12:50:50 SyncProducer [INFO] Disconnecting from
> > 10.181.18.87:9082
> > 2015-12-14 12:51:22 SimpleConsumer [INFO] Reconnect due to socket error:
> > java.nio.channels.ClosedChannelException
> >
> > Sometimes, there is a variation like this:
> >
> > 2015-12-14 13:05:47 ClientUtils$ [WARN] Fetching topic metadata with
> > correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from
> > broker [id:6,host:10.181.18.193,port:9082] failed
> > java.nio.channels.ClosedChannelException
> > at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
> > at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
> > at
> >
> >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
> > at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
> > at
> >
> >
> org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
> > at
> >
> >
> org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
> > at
> >
> >
> org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
> > at
> >
> >
> org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
> > at
> >
> >
> org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
> > at
> >
> >
> org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
> > at
> >
> >
> org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
> > at
> >
> >
> org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
> >
> > The above logs just keeps looping and the task never starts processing
> > input. I was able to telnet into the host/port combination from the same
> > machine. Any idea/pointers to what could be going wrong is greatly
> > appreciated.
> >
> > Thanks,
> >
> > KN.
> >
>



-- 
It is our choices that show what we truly are,
far more than our abilities.

Re: Review Request 41365: SAMZA-838: negative rocksdb.ttl.ms is not handled correctly

2015-12-15 Thread Tao Feng 


> On Dec. 15, 2015, 7:59 p.m., Navina Ramesh wrote:
> > samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala,
> >  line 54
> > 
> >
> > I don't think mentioning RocksDB wiki is necessary. You can change it 
> > to:  
> > "Non-positive TTL for RocksDB implies infinite TTL for the data."

Thanks Navina for the suggestion. I will update the rb.


- Tao


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41365/#review110553
---


On Dec. 14, 2015, 9:04 p.m., Tao Feng wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41365/
> ---
> 
> (Updated Dec. 14, 2015, 9:04 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> rocskdb ttl is handled per RocksDB 
> semantics(https://github.com/facebook/rocksdb/wiki/Time-to-Live) which zero 
> or negative ttl is same as infinity ttl.
> 
> 
> Diffs
> -
> 
>   
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala
>  211fc3be1e168f1f92812406785b39b5a3fd9174 
> 
> Diff: https://reviews.apache.org/r/41365/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew clean build &&  ./gradlew checkstyleMain checkstyleTest
> 
> 
> Thanks,
> 
> Tao Feng
> 
>

Re: Review Request 41365: SAMZA-838: negative rocksdb.ttl.ms is not handled correctly

2015-12-15 Thread Tao Feng 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41365/
---

(Updated Dec. 15, 2015, 11:58 p.m.)


Review request for samza.


Repository: samza


Description (updated)
---

update per Navina's comment


Diffs (updated)
-

  
samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala
 211fc3be1e168f1f92812406785b39b5a3fd9174 

Diff: https://reviews.apache.org/r/41365/diff/


Testing
---

./gradlew clean build &&  ./gradlew checkstyleMain checkstyleTest


Thanks,

Tao Feng

Re: Random connection errors

2015-12-15 Thread Kishore N C 
I did manage to locate a task which was blocked without starting as I had
described earlier. The log for that task is here:

https://gist.githubusercontent.com/kishorenc/ef806e85478378ce2203/raw/8537d3d3644eea7cdd9efc9fa8749c0840092f3c/gistfile1.txt

Thanks,

Kishore.

On Tue, Dec 15, 2015 at 4:30 PM, Kishore N C  wrote:

> Hi Yi Pan,
>
> I'm using Samza 0.9.1 and Kafka 0.8.2.1. Here's an example of a full task
> log:
>
>
> https://gist.githubusercontent.com/kishorenc/5d65f114a50b9ef6a6b3/raw/5b9ecffdd1af831f713e8b41e5b77e5b881e8173/gistfile1.txt
>
> You will find "java.nio.channels.ClosedChannelException" towards the end.
>
> One additional thing to mention here is that I detect when the Samza job
> has completed (by checking offsets periodically in the window() calls) and
> issue a "taskCoordinator.shutdown(RequestScope.CURRENT_TASK)" in my code.
>
> In some other cases, an error like:
>
> "2015-12-15 10:09:54 ClientUtils$ [WARN] Fetching topic metadata with
> correlation id 10 for topics [Set(TOPIC_TWO)] from broker
> [id:2,host:10.181.18.171,port:9082] failed
> java.nio.channels.ClosedChannelException"
>
> occurs right at the start of the job and the task would just refuse to
> start. Unfortunately, I don't have the logs for such a container anymore.
>
> Thanks,
>
> Kishore.
>
> On Tue, Dec 15, 2015 at 12:31 AM, Yi Pan  wrote:
>
>> Hi, Kishore,
>>
>> First, I would like to ask which version of Samza you are running? And if
>> you can attach the log and config of your container (i.e. I assume the log
>> you attached here is a container log?), it would be greatly helpful.
>>
>> Thanks a lot!
>>
>> -Yi
>>
>> On Mon, Dec 14, 2015 at 5:07 AM, Kishore N C  wrote:
>>
>> > Hi,
>> >
>> > I have a 25 node Samza cluster and I am running a job on a dataset of a
>> > billion records that is backed by a 7 node Kafka cluster.
>> >
>> > Some of the tasks on some of the Samza nodes don't seem to start at all
>> > (while other tasks run fine on other nodes). The specific error message
>> I
>> > see is in the task log is:
>> >
>> > 2015-12-14 12:50:50 ClientUtils$ [INFO] Fetching metadata from broker
>> > id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s)
>> > Set(TOPIC_ONE, TOPIC_TWO)
>> > 2015-12-14 12:50:50 SyncProducer [INFO] Connected to 10.181.18.87:9082
>> for
>> > producing
>> > 2015-12-14 12:50:50 SyncProducer [INFO] Disconnecting from
>> > 10.181.18.87:9082
>> > 2015-12-14 12:51:22 SimpleConsumer [INFO] Reconnect due to socket error:
>> > java.nio.channels.ClosedChannelException
>> >
>> > Sometimes, there is a variation like this:
>> >
>> > 2015-12-14 13:05:47 ClientUtils$ [WARN] Fetching topic metadata with
>> > correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from
>> > broker [id:6,host:10.181.18.193,port:9082] failed
>> > java.nio.channels.ClosedChannelException
>> > at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
>> > at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
>> > at
>> >
>> >
>> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
>> > at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
>> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
>> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
>> > at
>> >
>> >
>> org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
>> > at
>> >
>> >
>> org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
>> > at
>> >
>> >
>> org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
>> > at
>> >
>> >
>> org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
>> > at
>> >
>> >
>> org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
>> > at
>> >
>> >
>> org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
>> > at
>> >
>> >
>> org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
>> > at
>> >
>> >
>> org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
>> >
>> > The above logs just keeps looping and the task never starts processing
>> > input. I was able to telnet into the host/port combination from the same
>> > machine. Any idea/pointers to what could be going wrong is greatly
>> > appreciated.
>> >
>> > Thanks,
>> >
>> > KN.
>> >
>>
>
>
>
> --
> It is our choices that show what we truly are,
> far more than our abilities.
>



-- 
It is our choices that show what we truly are,
far more than our abilities.