killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-171475
> Sorry for the delay, I got airflow on kube set up on my laptop, so I'm
going to pick up this issue and attempt to implement option 1 .
>
> > First, stop the
killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1689250611
Hi, @yatink I did implement the k way merge suggested (although there were a
few issues with my implementation). I didn't notice that there was a resource [
killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1627849348
> Hmm, does it mean you’re doing the stream merge on the client (browser)
side?
Currently the merging is happening in the server side but it reads all the
logs into
killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1627826299
Hey @getaaron , I have implemented the k-way-merge and it works fine. For
now, I have specified the limit as 50 lines per call and it keeps calling till
the end_of_log is
killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1627823172
Hey @getaaron , I have implemented the k-way-merge and it works fine. For
now, I have specified the limit as 50 lines per call and it keeps calling till
the end_of_log is
killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1624548433
Thank you! I've started working on it, but I will reach out if I need any
help.
--
This is an automated message from the Apache Git Service.
To respond to the message,
killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1605737022
We can assign hard-limits and that would solve this current issue. We can
just sum filesizes before loading any of them (or most of them, I do not know
we can get all of
killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1605020397
Yeah, I thought of a way where we could maintain separate log_pos for
different streams. This way we do not have to pull the whole file into memory
just to get the final
killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1604974487
@potiuk The current log-streaming logic pulls the whole file into memory
before moving to the specific log_pos based on the metadata after combining all
the logs and
