Which method is throwing the exception? Is it stream.read()? If so, then I
would guess that the stream.read() implementation is caching a lot of data
in memory.
In any event, I don't think there's enough information in your code to
determine the cause of your problem. You probably need to use a profiler to
determine what's going on.
Brian MAso
On Thu, Nov 8, 2018 at 1:37 AM <4915374...@gmail.com> wrote:
> Hello,Every Hacker:
> I am a junior software engineer. I tried to rewrite Stream as a sink and
> found that when I read a large update of the database, I throw an exception
> OutOfMemory. Can anyone give me some advice? Why am I wrong, thanks :-)
>
> def run(url: String, user: String, password: String, replUser: String,
> replPass: String): Unit = {
> implicit val system: ActorSystem = ActorSystem("log-analyzer")
> implicit val materializer: ActorMaterializer = ActorMaterializer()
> implicit val ec: ExecutionContextExecutor = system.dispatcher
>
>
> class DeadLetterListener extends Actor {
> def receive: PartialFunction[Any, Unit] = {
> case dl: DeadLetter ⇒
> logger.error("dead letter: {}", dl)
> }
> }
>
> val listener = system.actorOf(Props[DeadLetterListener])
> system.eventStream.subscribe(listener, classOf[DeadLetter])
>
> Class.forName("org.postgresql.Driver")
>
> // check last confirmed lsn
> val sqlConnection: Connection = this.getSqlConnection(url, user, password)
> val lastLSNOpt: Option[LogSequenceNumber] =
> getLastConfirmedLSN(SLOT_NAME).run(sqlConnection)
>
> var stream: PGReplicationStream = null
> var replConnection: PGConnection = this.getReplicationConnection(url,
> replUser, replPass)
>
> if (lastLSNOpt.isEmpty) {
> logger.info(s"未发现slot信息，初始化slot: $SLOT_NAME")
> createSlot(replConnection)
> } else {
> logger.info(s"从 ${lastLSNOpt.get} 开始")
> }
>
>
> // start stream flow process
> // val counter = new AtomicInteger(0)
>
> var lastTs = System.currentTimeMillis()
> var lastReceiveLSN = LogSequenceNumber.INVALID_LSN
>
> while (true) {
> try {
> if (replConnection == null ||
> replConnection.asInstanceOf[Connection].isClosed) {
> replConnection = this.getReplicationConnection(url, replUser,
> replPass)
> }
>
> stream = replConnection
> .getReplicationAPI
> .replicationStream
> .logical
> .withSlotName(SLOT_NAME)
> .withSlotOption("include-xids", false)
> // .withSlotOption("skip-empty-xacts", true)
> test_decoding下的设定，wal2json不适用
> .withStatusInterval(STATUS_INTERVAL.toSeconds.asInstanceOf[Int],
> TimeUnit.SECONDS)
> .withStartPosition(lastReceiveLSN)
> .start() // start replication stream
>
>
> object stream2iter extends Iterator[(LogSequenceNumber, String)] {
> override def next: (LogSequenceNumber, String) = {
> try {
> //outofmemory when big update in database
> val msg = stream.read
> //println(msg)
> val offset = msg.arrayOffset
> val src = msg.array
> val len = src.length - offset
> val log = new String(src, offset, len)
> logger.debug(log)
> val tmpLSN = stream.getLastReceiveLSN
>
> (tmpLSN, log)
> } catch {
> case e: Throwable =>
> println("localhost: " + e)
> (LogSequenceNumber.INVALID_LSN, "")
> }
> }
>
> override def hasNext: Boolean = !stream.isClosed
> }
>
> val srcStream: Source[(LogSequenceNumber, String), NotUsed] =
> Source.fromIterator(() => stream2iter)
>
>
> val sink: Sink[LogRecord, Future[Done]] = Sink.foreach(log => {
> logger.debug(s"LSN: ${log.lsn}, target: ${log.target.fullname}")
> if (log.lsn != LogSequenceNumber.INVALID_LSN)
> lastReceiveLSN = log.lsn
> stream.setAppliedLSN(lastReceiveLSN)
> stream.setFlushedLSN(lastReceiveLSN)
>
> })
>
> val totalG: RunnableGraph[NotUsed] =
> srcStream.via(graph(system)).to(sink)
> val res = totalG.run()
> } catch {
> // 断连常态化
> case e: PSQLException =>
> logger.warn("Connection closed, retrying...", e)
> val conn = replConnection.asInstanceOf[Connection]
> if (!conn.isClosed) {
> conn.close()
> }
> TimeUnit.SECONDS.sleep(1)
> restarts.inc()
> replConnection = null
> case e: Throwable =>
> logger.error("Connection can not be established!", e)
> shutdown(system, e)
> }
> }
>
> }
>
> --
>
> *****************************************************************************************************
> ** New discussion forum: https://discuss.akka.io/ replacing akka-user
> google-group soon.
> ** This group will soon be put into read-only mode, and replaced by
> discuss.akka.io
> ** More details:
> https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
>
> *****************************************************************************************************
> >>>>>>>>>>
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user
google-group soon.
** This group will soon be put into read-only mode, and replaced by
discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************
>>>>>>>>>>
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
