As a part of my PhD research, I have been working on few optimization algorithms which try to jointly optimize delay and traffic (WAN traffic) in a geo-distributed streaming analytics setting. So, to show that the optimization actually works in real life, I am trying to implement these optimization algorithms on top of Apache Flink. For emulating a real life example, I need to generate a stream of records with some realistic delay (order of microseconds for fast incoming stream) between any two records. This stream will then by ingested and processed by Flink.
Using the timestamps as is, in the form of event timestamps, only proves the algorithms from a theoretical/simulation perspective. Hope this answers your question to some extent at least. Let me know. Thanks! -------------------------------------------------- Dhruv Kumar PhD Candidate Department of Computer Science and Engineering University of Minnesota www.dhruvkumar.me > On May 15, 2018, at 20:29, Xingcan Cui <xingc...@gmail.com> wrote: > > Hi Dhruv, > > since there are timestamps associated with each record, I was wondering why > you try to replay them with a fixed interval. Can you give a little > explanation about that? > > Thanks, > Xingcan > >> On May 16, 2018, at 2:11 AM, Ted Yu <yuzhih...@gmail.com >> <mailto:yuzhih...@gmail.com>> wrote: >> >> Please see the following: >> >> http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html >> <http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html> >> >> https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds >> >> <https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds> >> >> On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar <gargdhru...@gmail.com >> <mailto:gargdhru...@gmail.com>> wrote: >> Hi >> >> I am trying to replay a log file in which each record has a timestamp >> associated with it. The time difference between the records is of the order >> of microseconds. I am trying to replay this log maintaining the same delay >> between the records (using Thread.sleep()) and sending it to a socket. And >> then the Flink program reads the incoming data from this socket. Currently, >> replay of the entire log file takes much more time (3 times) then the >> expected time (last_timstamp - first_timstamp). >> >> I wanted to know what are the standard ways of replaying log files if one >> wants to maintain the same arrival delay between the records. >> >> Let me know if I am not clear above. >> >> Thanks >> -------------------------------------------------- >> Dhruv Kumar >> PhD Candidate >> Department of Computer Science and Engineering >> University of Minnesota >> www.dhruvkumar.me <http://www.dhruvkumar.me/> >> >
