[GitHub] [gobblin] phet commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

phet commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1082123322 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java: ## @@ -98,18 +102,20 @@ protected void processMessage(DecodeableKa

[GitHub] [gobblin] codecov-commenter commented on pull request #3627: Specifically name each Hikari connection pool created, for traceability

codecov-commenter commented on PR #3627: URL: https://github.com/apache/gobblin/pull/3627#issuecomment-1397906097 # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3627?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Sof

[GitHub] [gobblin] phet opened a new pull request, #3627: Specifically name each Hikari connection pool created, for traceability

phet opened a new pull request, #3627: URL: https://github.com/apache/gobblin/pull/3627 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the following [

[GitHub] [gobblin] ZihanLi58 commented on a diff in pull request #3624: [GOBBLIN-1765] Add support to sync metadata for dir in manifest based copy

ZihanLi58 commented on code in PR #3624: URL: https://github.com/apache/gobblin/pull/3624#discussion_r1081946270 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/CopyableFile.java: ## @@ -316,23 +316,33 @@ private List replicateAncestorsOwnerAndP

[GitHub] [gobblin] Will-Lo merged pull request #3614: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs

Will-Lo merged PR #3614: URL: https://github.com/apache/gobblin/pull/3614 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache

[GitHub] [gobblin] umustafi commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1081842299 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java: ## @@ -176,6 +185,7 @@ protected void createMetrics() { t

[GitHub] [gobblin] umustafi commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1081826513 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java: ## @@ -98,18 +102,20 @@ protected void processMessage(Decodeab

[GitHub] [gobblin] umustafi commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1081821627 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/metrics/RuntimeMetrics.java: ## @@ -36,15 +36,17 @@ public class RuntimeMetrics { public static final St

[GitHub] [gobblin] Will-Lo merged pull request #3471: [GOBBLIN-1612] Add description about downloading gradle wrapper

Will-Lo merged PR #3471: URL: https://github.com/apache/gobblin/pull/3471 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache

[GitHub] [gobblin] phet commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

phet commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1081820136 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java: ## @@ -333,4 +333,8 @@ public void run() { } } } + + public Long

[GitHub] [gobblin] Will-Lo merged pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo merged PR #3623: URL: https://github.com/apache/gobblin/pull/3623 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache

[GitHub] [gobblin] umustafi commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1081680380 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java: ## @@ -333,4 +333,8 @@ public void run() { } } } + + public L

[GitHub] [gobblin] umustafi commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1081645025 ## gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GenericStoreChangeEvent.avsc: ## @@ -9,9 +9,14 @@ "doc" : "Primary key for the store", "compliance

[GitHub] [gobblin] phet commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

phet commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1081058477 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java: ## @@ -333,4 +333,8 @@ public void run() { } } } + + public Long

[GitHub] [gobblin] phet commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

phet commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1081035824 ## gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/runtime/KafkaAvroJobStatusMonitorTest.java: ## @@ -618,6 +728,7 @@ public org.apache.gobblin.configura

[GitHub] [gobblin] phet commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

phet commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1081033861 ## gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/runtime/KafkaAvroJobStatusMonitorTest.java: ## @@ -358,6 +370,66 @@ public void testProcessingRetried

[GitHub] [gobblin] phet commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

phet commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1081029179 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [gobblin] phet commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

phet commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1081029179 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [gobblin] umustafi commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1080737848 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java: ## @@ -98,18 +102,20 @@ protected void processMessage(Decodeab

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1080707760 ## gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/runtime/KafkaAvroJobStatusMonitorTest.java: ## @@ -527,13 +637,14 @@ private GobblinTrackingEvent c

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1080707399 ## gobblin-service/src/test/java/org/apache/gobblin/service/monitoring/GaaSObservabilityProducerTest.java: ## @@ -0,0 +1,117 @@ +/* + * Licensed to the Apache Software

[GitHub] [gobblin] phet commented on a diff in pull request #3624: [GOBBLIN-1765] Add support to sync metadata for dir in manifest based copy

phet commented on code in PR #3624: URL: https://github.com/apache/gobblin/pull/3624#discussion_r1080643334 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/CopyableFile.java: ## @@ -316,23 +316,33 @@ private List replicateAncestorsOwnerAndPermis

[GitHub] [gobblin] phet commented on a diff in pull request #3624: [GOBBLIN-1765] Add support to sync metadata for dir in manifest based copy

phet commented on code in PR #3624: URL: https://github.com/apache/gobblin/pull/3624#discussion_r1080640043 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/ManifestBasedDataset.java: ## @@ -125,9 +123,31 @@ public Iterator> getFileSetIterator(Fi

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1080655252 ## gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/runtime/KafkaAvroJobStatusMonitorTest.java: ## @@ -387,62 +460,95 @@ public void testProcessProgre

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1080652649 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software

[GitHub] [gobblin] ZihanLi58 commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

ZihanLi58 commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1080626419 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java: ## @@ -288,6 +303,15 @@ private static void modifyStateIfRetryRequ

[GitHub] [gobblin] umustafi commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1080610719 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/SpecStoreChangeMonitor.java: ## @@ -171,5 +178,6 @@ protected void createMetrics() { this.d

[GitHub] [gobblin] umustafi commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1080604384 ## gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GenericStoreChangeEvent.avsc: ## @@ -9,9 +9,14 @@ "doc" : "Primary key for the store", "compliance

[GitHub] [gobblin] umustafi commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1080602171 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java: ## @@ -98,18 +102,20 @@ protected void processMessage(Decodeab

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1080592247 ## gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/runtime/KafkaAvroJobStatusMonitorTest.java: ## @@ -358,6 +370,66 @@ public void testProcessingRetr

[GitHub] [gobblin] phet commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

phet commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1080573520 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [gobblin] phet commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

phet commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1080567642 ## gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/runtime/KafkaAvroJobStatusMonitorTest.java: ## @@ -387,62 +460,95 @@ public void testProcessProgressi

[GitHub] [gobblin] umustafi commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1080587369 ## gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GenericStoreChangeEvent.avsc: ## @@ -9,9 +9,14 @@ "doc" : "Primary key for the store", "compliance

[GitHub] [gobblin] AndyJiang99 commented on a diff in pull request #3614: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs

AndyJiang99 commented on code in PR #3614: URL: https://github.com/apache/gobblin/pull/3614#discussion_r1080581336 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/dataset/FSVolumeDatasetDescriptor.java: ## @@ -19,38 +19,41 @@ import com.typesafe.config.Con

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3614: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs

Will-Lo commented on code in PR #3614: URL: https://github.com/apache/gobblin/pull/3614#discussion_r1080575218 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/dataset/FSVolumeDatasetDescriptor.java: ## @@ -19,38 +19,41 @@ import com.typesafe.config.Config;

[GitHub] [gobblin] Will-Lo merged pull request #3626: [GOBBLIN-1767] Update references to deprecated Mysql connector/j driver to new name

Will-Lo merged PR #3626: URL: https://github.com/apache/gobblin/pull/3626 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1073873739 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software

[GitHub] [gobblin] codecov-commenter commented on pull request #3626: [GOBBLIN-1767] Update references to deprecated Mysql connector/j driver to new name

codecov-commenter commented on PR #3626: URL: https://github.com/apache/gobblin/pull/3626#issuecomment-1386640426 # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3626?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Sof

[GitHub] [gobblin] phet opened a new pull request, #3626: [GOBBLIN-1767] Update references to deprecated Mysql connector/j driver to new name

phet opened a new pull request, #3626: URL: https://github.com/apache/gobblin/pull/3626 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the following [

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1073005648 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java: ## @@ -65,9 +65,9 @@ public class KafkaAvroJobStatusMonitor exten

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1073002292 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1073002292 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1073001902 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software

[GitHub] [gobblin] phet commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

phet commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1072969190 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/SpecStoreChangeMonitor.java: ## @@ -171,5 +178,6 @@ protected void createMetrics() { this.delet

[GitHub] [gobblin] phet commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

phet commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1072949890 ## gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GenericStoreChangeEvent.avsc: ## @@ -9,9 +9,14 @@ "doc" : "Primary key for the store", "compliance" :

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1072967326 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1072966969 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java: ## @@ -219,7 +229,8 @@ protected void processMessage(DecodeableKafka

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1072966717 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java: ## @@ -136,6 +140,12 @@ public void onRetry(Attempt attempt) {

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1072965455 ## gobblin-service/src/test/java/org/apache/gobblin/service/monitoring/GaaSObservabilityProducerTest.java: ## @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1072965138 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/troubleshooter/JobIssueEventHandler.java: ## @@ -50,6 +51,7 @@ public class JobIssueEventHandler { privat

[GitHub] [gobblin] ZihanLi58 commented on a diff in pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

ZihanLi58 commented on code in PR #3625: URL: https://github.com/apache/gobblin/pull/3625#discussion_r1072860103 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java: ## @@ -98,18 +102,20 @@ protected void processMessage(Decodea

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3614: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs

Will-Lo commented on code in PR #3614: URL: https://github.com/apache/gobblin/pull/3614#discussion_r1072710897 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/flowgraph/DatasetDescriptorErrorUtils.java: ## @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Soft

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3614: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs

Will-Lo commented on code in PR #3614: URL: https://github.com/apache/gobblin/pull/3614#discussion_r1072709823 ## gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigV2ResourceLocalHandler.java: ## @@ -

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3614: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs

Will-Lo commented on code in PR #3614: URL: https://github.com/apache/gobblin/pull/3614#discussion_r1068845916 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/template/StaticFlowTemplate.java: ## @@ -56,6 +57,8 @@ @Slf4j public class StaticFlowTemplate impl

[GitHub] [gobblin] arjun4084346 commented on pull request #3623: [GOBBLIN-1764] Emit observability event

arjun4084346 commented on PR #3623: URL: https://github.com/apache/gobblin/pull/3623#issuecomment-1385907611 I read the description of the PR, can you please elaborate more what benefit GaaSObservabilityEvents has over a GobblinTrackingEvent with name="GaaSObservabilityEvent". How is it mor

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1072534871 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java: ## @@ -288,6 +303,15 @@ private static void modifyStateIfRetryRequir

[GitHub] [gobblin] codecov-commenter commented on pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

codecov-commenter commented on PR #3625: URL: https://github.com/apache/gobblin/pull/3625#issuecomment-1382641303 # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3625?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Sof

[GitHub] [gobblin] umustafi commented on pull request #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi commented on PR #3625: URL: https://github.com/apache/gobblin/pull/3625#issuecomment-1382638995 @ZihanLi58 @phet can you review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [gobblin] umustafi opened a new pull request, #3625: [GOBBLIN-1766] Define metric to measure lag from producing to consume…

umustafi opened a new pull request, #3625: URL: https://github.com/apache/gobblin/pull/3625 … change stream events Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [

[GitHub] [gobblin] codecov-commenter commented on pull request #3624: [GOBBLIN-1765] Add support to sync metadata for dir in manifest based copy

codecov-commenter commented on PR #3624: URL: https://github.com/apache/gobblin/pull/3624#issuecomment-1382634074 # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3624?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Sof

[GitHub] [gobblin] ZihanLi58 opened a new pull request, #3624: [GOBBLIN-1765] Add support to sync metadata for dir in manifest based copy

ZihanLi58 opened a new pull request, #3624: URL: https://github.com/apache/gobblin/pull/3624 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the follow

[GitHub] [gobblin] homatthew commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

homatthew commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1070151228 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java: ## @@ -219,7 +229,8 @@ protected void processMessage(DecodeableKaf

[GitHub] [gobblin] homatthew commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

homatthew commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1070151228 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java: ## @@ -219,7 +229,8 @@ protected void processMessage(DecodeableKaf

[GitHub] [gobblin] phet commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

phet commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1070103059 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [gobblin] phet commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

phet commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1070071056 ## gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/runtime/KafkaAvroJobStatusMonitorTest.java: ## @@ -89,6 +97,7 @@ public class KafkaAvroJobStatusMonito

[GitHub] [gobblin] phet commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

phet commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1070071056 ## gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/runtime/KafkaAvroJobStatusMonitorTest.java: ## @@ -89,6 +97,7 @@ public class KafkaAvroJobStatusMonito

[GitHub] [gobblin] ZihanLi58 commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

ZihanLi58 commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1070086617 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GaaSObservabilityEventProducer.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Softwa

[GitHub] [gobblin] phet commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

phet commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1070057390 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java: ## @@ -219,7 +229,8 @@ protected void processMessage(DecodeableKafkaRec

[GitHub] [gobblin] homatthew commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

homatthew commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1070031403 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java: ## @@ -219,7 +229,8 @@ protected void processMessage(DecodeableKaf

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

Will-Lo commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1070030849 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java: ## @@ -266,6 +277,11 @@ static void addJobStatusToStateStore(org.apa

[GitHub] [gobblin] homatthew commented on a diff in pull request #3623: [GOBBLIN-1764] Emit observability event

homatthew commented on code in PR #3623: URL: https://github.com/apache/gobblin/pull/3623#discussion_r1070022305 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java: ## @@ -266,6 +277,11 @@ static void addJobStatusToStateStore(org.a

[GitHub] [gobblin] codecov-commenter commented on pull request #3623: [GOBBLIN-1764] Emit observability event

codecov-commenter commented on PR #3623: URL: https://github.com/apache/gobblin/pull/3623#issuecomment-1382257855 # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3623?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Sof

[GitHub] [gobblin] phet commented on a diff in pull request #3616: [GOBBLIN-1755] Support extended ACLs and sticky bit for file based distcp

phet commented on code in PR #3616: URL: https://github.com/apache/gobblin/pull/3616#discussion_r1069857107 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/PreserveAttributes.java: ## @@ -45,6 +45,7 @@ public static enum Option { GROUP('g'),

[GitHub] [gobblin] Will-Lo opened a new pull request, #3623: [GOBBLIN-1764] Emit observability event

Will-Lo opened a new pull request, #3623: URL: https://github.com/apache/gobblin/pull/3623 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the followin

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3616: [GOBBLIN-1755] Support extended ACLs and sticky bit for file based distcp

Will-Lo commented on code in PR #3616: URL: https://github.com/apache/gobblin/pull/3616#discussion_r1069681531 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/writer/FileAwareInputStreamDataWriter.java: ## @@ -500,15 +504,26 @@ private void ensur

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3614: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs

Will-Lo commented on code in PR #3614: URL: https://github.com/apache/gobblin/pull/3614#discussion_r1068619264 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/template/StaticFlowTemplate.java: ## @@ -165,19 +166,38 @@ private Config getResolvedFlowConfig(Conf

[GitHub] [gobblin] Will-Lo merged pull request #3621: [GOBBLIN-1762] Upgrade Gobblin OSS Hadoop version to 2.10.0

Will-Lo merged PR #3621: URL: https://github.com/apache/gobblin/pull/3621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache

[GitHub] [gobblin] phet commented on a diff in pull request #3616: [GOBBLIN-1755] Support extended ACLs and sticky bit for file based distcp

phet commented on code in PR #3616: URL: https://github.com/apache/gobblin/pull/3616#discussion_r1067725962 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/PreserveAttributes.java: ## @@ -45,6 +45,8 @@ public static enum Option { GROUP('g'),

[GitHub] [gobblin] ZihanLi58 merged pull request #3603: [GOBBLIN-1744] Improve handling of null value edge cases when querying Helix

ZihanLi58 merged PR #3603: URL: https://github.com/apache/gobblin/pull/3603 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apac

[GitHub] [gobblin] homatthew commented on a diff in pull request #3603: [GOBBLIN-1744] Improve handling of null value edge cases when querying Helix

homatthew commented on code in PR #3603: URL: https://github.com/apache/gobblin/pull/3603#discussion_r106638 ## gobblin-cluster/src/main/java/org/apache/gobblin/cluster/HelixUtils.java: ## @@ -395,11 +392,22 @@ private static void deleteStoppedHelixJob(HelixManager helixMan

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3614: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs

Will-Lo commented on code in PR #3614: URL: https://github.com/apache/gobblin/pull/3614#discussion_r1066380678 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/dataset/BaseDatasetDescriptor.java: ## @@ -57,37 +63,44 @@ public BaseDatasetDescriptor(Config confi

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3614: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs

Will-Lo commented on code in PR #3614: URL: https://github.com/apache/gobblin/pull/3614#discussion_r1066038913 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/template/StaticFlowTemplate.java: ## @@ -165,19 +164,40 @@ private Config getResolvedFlowConfig(Conf

[GitHub] [gobblin] Will-Lo commented on pull request #3609: [GOBBLIN-1749] Add dependency for handling xz-compressed Avro file

Will-Lo commented on PR #3609: URL: https://github.com/apache/gobblin/pull/3609#issuecomment-1377544151 Thank you @sekikn for the contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [gobblin] Will-Lo merged pull request #3622: [GOBBLIN-1763] D2 markup/down for all live GaaS services not only leader

Will-Lo merged PR #3622: URL: https://github.com/apache/gobblin/pull/3622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache

[GitHub] [gobblin] codecov-commenter commented on pull request #3622: D2 markup/down for all live GaaS services not only leader

codecov-commenter commented on PR #3622: URL: https://github.com/apache/gobblin/pull/3622#issuecomment-1376144422 # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3622?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Sof

[GitHub] [gobblin] umustafi opened a new pull request, #3622: D2 markup/down for all live GaaS services not only leader

umustafi opened a new pull request, #3622: URL: https://github.com/apache/gobblin/pull/3622 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [X] My PR addresses the followi

[GitHub] [gobblin] meethngala opened a new pull request, #3621: [GOBBLIN-1762] Upgrade Gobblin OSS Hadoop version to 2.10.0

meethngala opened a new pull request, #3621: URL: https://github.com/apache/gobblin/pull/3621 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the follo

[GitHub] [gobblin] Will-Lo merged pull request #3620: [GOBBLIN-1761] Update Gobblin OSS Slack channel link to a never-expire link

Will-Lo merged PR #3620: URL: https://github.com/apache/gobblin/pull/3620 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3614: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs

Will-Lo commented on code in PR #3614: URL: https://github.com/apache/gobblin/pull/3614#discussion_r1063865584 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/dataset/FSDatasetPartitionConfig.java: ## @@ -128,13 +134,19 @@ private void validatePartitionConfig

[GitHub] [gobblin] codecov-commenter commented on pull request #3620: [GOBBLIN-1761] Update Gobblin OSS Slack channel link to a never-expire link

codecov-commenter commented on PR #3620: URL: https://github.com/apache/gobblin/pull/3620#issuecomment-1374222141 # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3620?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Sof

[GitHub] [gobblin] Will-Lo opened a new pull request, #3620: [GOBBLIN-1761] Update Gobblin OSS Slack channel link to a never-expire link

Will-Lo opened a new pull request, #3620: URL: https://github.com/apache/gobblin/pull/3620 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the followin

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3614: Add error reporting when attempting to resolve flow configs

Will-Lo commented on code in PR #3614: URL: https://github.com/apache/gobblin/pull/3614#discussion_r1062687327 ## gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigV2ResourceLocalHandler.java: ## @@ -

[GitHub] [gobblin] Will-Lo merged pull request #3610: [GOBBLIN-1750] Add schemas for observability events in GaaS

Will-Lo merged PR #3610: URL: https://github.com/apache/gobblin/pull/3610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache

[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3610: [GOBBLIN-1750] Add schemas for observability events in GaaS

Will-Lo commented on code in PR #3610: URL: https://github.com/apache/gobblin/pull/3610#discussion_r1060789952 ## gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GaaSObservabilityEventExperimental.avsc: ## @@ -0,0 +1,167 @@ +{ + "type": "record", + "name": "GaaSObserva

[GitHub] [gobblin] codecov-commenter commented on pull request #3614: Andjiang/flow config error reporting

codecov-commenter commented on PR #3614: URL: https://github.com/apache/gobblin/pull/3614#issuecomment-1362255035 # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3614?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Sof

[GitHub] [gobblin] umustafi commented on a diff in pull request #3610: [GOBBLIN-1750] Add schemas for observability events in GaaS

umustafi commented on code in PR #3610: URL: https://github.com/apache/gobblin/pull/3610#discussion_r1054899586 ## gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GaaSObservabilityEventExperimental.avsc: ## @@ -0,0 +1,167 @@ +{ + "type": "record", + "name": "GaaSObserv

[GitHub] [gobblin] ZihanLi58 merged pull request #3618: [GOBBLIN-1757]Refactor manifest, add reader/writer and iterator for efficient reading

ZihanLi58 merged PR #3618: URL: https://github.com/apache/gobblin/pull/3618 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apac

[GitHub] [gobblin] mccormickt12 commented on a diff in pull request #3618: [GOBBLIN-1757]Refactor manifest, add reader/writer and iterator for efficient reading

mccormickt12 commented on code in PR #3618: URL: https://github.com/apache/gobblin/pull/3618#discussion_r1053789735 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/CopyManifest.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [gobblin] ZihanLi58 merged pull request #3619: [GOBBLIN-1758] Disable flaky HiveMaterializerTest on CI/CD

ZihanLi58 merged PR #3619: URL: https://github.com/apache/gobblin/pull/3619 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apac

[GitHub] [gobblin] ZihanLi58 commented on a diff in pull request #3618: [GOBBLIN-1757]Refactor manifest, add reader/writer and iterator for efficient reading

ZihanLi58 commented on code in PR #3618: URL: https://github.com/apache/gobblin/pull/3618#discussion_r1053780279 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/CopyManifest.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Found

  1   2   3   4   5   6   7   8   9   10   >