schizophrenicmaniac opened a new pull request, #40481: URL: https://github.com/apache/beam/pull/40481
Adds a SchemaTransform for SplunkIO writes and exposes it in Beam YAML as `WriteToSplunk`. It follows the same approach as the Datadog one (#38362). SplunkIO in Java only supports writes, so there is no `ReadFromSplunk` here. Each input row becomes a `SplunkEvent`. `event` (string) is required. `time` (int64), `host`, `source`, `sourcetype`, `index` and `fields` are optional. `fields` can be a nested row or a JSON object string, and other columns are ignored. The config covers everything `SplunkIO.Write` supports (url, token, batch_count, parallelism, disable_certificate_validation, root_ca_certificate_path, enable_batch_logs, enable_gzip_http_compression) plus error_handling. The error output has the same shape as Datadog's: rows that can't be converted come back with `failed_row` and a 400, and HEC write failures come back with the HTTP status and the payload. Without error_handling, a failed write fails the pipeline. While testing this I found that `SplunkWriteError` loses all of its values once it goes through a coder. It has `@DefaultSchema(AutoValueSchema.class)`, but its accessors aren't `get`-prefixed, so the inferred schema has no fields and every error decodes as null/null/null. `SplunkEventWriterTest` doesn't catch this because `containsInAnyOrder` decodes both sides with the same coder. I added a `SplunkWriteErrorCoder` (same idea as `DatadogWriteErrorCoder`) and set it on the output of `SplunkIO.Write`. This changes the output coder of `SplunkIO.Write`, so I listed it under bugfixes in CHANGES.md. Changes: - `SplunkWriteSchemaTransformProvider` / `SplunkWriteSchemaTransformConfiguration`, registered in the IO expansion service - `WriteToSplunk` in standard_io.yaml and the matching entry in standard_external_transforms.yaml - `SplunkWriteErrorCoder`, used by `SplunkIO.Write` - Java tests against a mock HEC server, and a YAML integration test with a fake HEC fixture (tests/splunk.yaml) Fixes #40470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
