The key point here is, how do you jump to the log service from Spark UI to explore or download logs of each Pod like Spark on Yarn?
Thanks, Cheng Pan On Nov 2, 2022 at 18:32:26, Martin Andersson <martin.anders...@kambi.com> wrote: > Hello Cheng. > > I don't quite understand, why can't you configure Log4j bundled with spark > to write logs in whatever format you need, then use something like promtail > to export the logs to whatever log service you want to use? > > BR, > Martin > ------------------------------ > *From:* Cheng Pan <cheng...@apache.org> > *Sent:* Tuesday, November 1, 2022 17:03 > *To:* spark-dev <dev@spark.apache.org> > *Subject:* Allow Spark on K8s to integrate w/ External Log Service > > > EXTERNAL SENDER. Do not click links or open attachments unless you > recognize the sender and know the content is safe. DO NOT provide your > username or password. > > Hi Spark community, > > I’m recently working on migrating Spark workloads from Yarn to K8s, and > one of the issue I found is that there is no out-of-box log solution for > Spark applications on K8s as Yarn has, and Spark even does not provide a > way to integrate w/ external log service. > > Based on the above situation, I propose a way[1][2] to allow Spark to > integrate w/ external log service when running on K8s, and PR[3] is > available for reviewing and testing. > > I also found there was a PR[4] wants to the similar thing. > > Please let me know if you have any feedback on this idea, and other log > solutions/ideas/suggestions for Spark on K8s are welcome as well. > > [1] > https://docs.google.com/document/d/1MfB39LD4B4Rp7MDRxZbMKMbdNSe6V6mBmMQ-gkCnM-0/edit?usp=sharing > <https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1MfB39LD4B4Rp7MDRxZbMKMbdNSe6V6mBmMQ-gkCnM-0%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Cmartin.andersson%40kambi.com%7C3d098567685c47a3c1de08dabc22bdc0%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638029154642900019%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G7qXFK2msMyB5OI3xUwQ4LqkNnlMnc5SpzmMHApLE1o%3D&reserved=0> > [2] https://issues.apache.org/jira/browse/SPARK-40887 > <https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSPARK-40887&data=05%7C01%7Cmartin.andersson%40kambi.com%7C3d098567685c47a3c1de08dabc22bdc0%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638029154642900019%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7ssnO23zcDErOcZsi9I09aMNIxq5xrV0AIVQA8386VA%3D&reserved=0> > [3] https://github.com/apache/spark/pull/38357 > <https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fpull%2F38357&data=05%7C01%7Cmartin.andersson%40kambi.com%7C3d098567685c47a3c1de08dabc22bdc0%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638029154643056253%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UcOJUhZSKlL30QiTUWgGBIMiHuLrPSrBHa5sBf4vpVw%3D&reserved=0> > [4] https://github.com/apache/spark/pull/32456 > <https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fpull%2F32456&data=05%7C01%7Cmartin.andersson%40kambi.com%7C3d098567685c47a3c1de08dabc22bdc0%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638029154643056253%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OHNZqnEjwAH%2F2aEQTCxbqHJEwgt6hWc8cVSDahoIBkQ%3D&reserved=0> > > Thanks, > Cheng Pan >
