[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-884917062 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-884916961 > @AngersZh Thanks. Please update the screenshot in the PR description as well. DOne -- This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-883463577 > @AngersZh can you update the screenshot in the pr description? Done -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-883354294 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-882546959 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-882546959 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-882546959 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-882176352 Any more suggestion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-881127826 ping @cloud-fan @gengliangwang @maropu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-806307672 gentle ping @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-794964106 > We can periodically log something in the built-in file commit protocol, but there is nothing we can do if people are using a custom file commit protocol. A new

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-794853587 > And for the extreme case like taking hours on committing, I think more important thing is to log periodically to let end users determine whether the Spark driver is hang

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-793672126 > How often is that? We can also improve the log to make it easier to search for a certain job. another case , we run a MSCK of table, in SQL tab it shows nothing.

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-793484750 In fact, the cluster environment of many companies is not so healthy, and there are often slow nodes that cause the commit and hive metadata load table/partition to be

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-785554038 > Normally, committing a job should be fast. I don't think it is a good idea to put this in the SQL graph. For debug purposes, the log message should be enough. >

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-783854398 > The file commit is a driver side thing, why do we need to update `BasicWriteJobStatsTracker`? I think we can follow `BroadcastExchangeExec` and simply call

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-782549928 Gentle ping @HeartSaVioR @dongjoon-hyun @HyukjinKwon @maropu @cloud-fan Could you help to review this I think it's really help since always `INSERT` statement slow caused

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-775725455 Gentle ping @dongjoon-hyun @HeartSaVioR @HyukjinKwon This is an automated message from the Apache Git