alexey-bass opened a new issue, #13273: URL: https://github.com/apache/seatunnel/issues/13273
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened With a job whose serialized `JobImmutableInformation` is larger than about 16 MB (in our case a Postgres-CDC job with a `table-pattern` matching ~730 tables), `GET /running-jobs/summary` returns a response of tens of megabytes: `jobName` contains ~16 MB of raw serialized bytes. The web UI's Overview and Jobs pages load this endpoint and hang in the browser. `/running-jobs` and `/job-info/<id>` return correct data, and the job itself runs normally. **Cause** `JobInfoService.decodeJobBasicInfo` reads the fields directly from the `Data` buffer: ```java // IdentifiedDataSerializable header: factoryId + classId. in.readInt(); in.readInt(); in.readLong(); // jobId String jobName = in.readString(); in.readBoolean(); // isStartWithSavePoint long createTime = in.readLong(); ``` Hazelcast's `DataSerializableSerializer` writes a 1-byte header (identified / versioned flags) before `factoryId` and `classId`. The fast path does not skip it, so every following read is shifted by one byte. `readString()` then takes its length from the last byte of `jobId` followed by the first three bytes of the real length, i.e. `(jobId & 0xFF) << 24` for any normal job name. All job ids in our cluster end in `0x01`, so the length is 16,777,216. - Serialized job smaller than that: `readString` hits EOF, the `catch (Throwable)` fallback does the full deserialization and the result is correct. This hides the bug for normal jobs (and means the fast path never actually takes effect). - Serialized job larger than that: `readString` succeeds and returns 16 MB of the buffer as `jobName`; `createTime` is garbage too. **Expected** `jobName` and `createTime` of the job; a response of a few hundred bytes per job. **How to reproduce** Submit any job whose `JobImmutableInformation` exceeds 16 MB (for example a CDC source with several hundred wide tables), then call `GET /running-jobs/summary` or open the UI Overview page. **Suggested fix** Skip the header byte before the two ints (`in.readByte()`), and additionally validate the decoded name (for example, fall back to full deserialization when the length is implausible). A unit test with a `JobImmutableInformation` larger than 16 MB would cover it. ### SeaTunnel Version 3.0.0 (the method is unchanged on `dev`) ### SeaTunnel Config ```conf # shape of the job; names and hosts replaced env { job.name = "cdc_example" job.mode = "STREAMING" parallelism = 1 checkpoint.interval = 30000 } source { Postgres-CDC { plugin_output = "pg_cdc" url = "jdbc:postgresql://<host>:5432/<db>" username = "<user>" password = "<password>" database-names = ["<db>"] # matches ~730 daily tables with ~54 columns each table-pattern = "<db>\\.(partition\\.event_([0-9]{4}-[0-9]{2}-[0-9]{2}))" database-pattern = "<db>" decoding.plugin.name = "pgoutput" slot.name = "<slot>" startup.mode = "latest" } } transform { TableMerge { plugin_input = "pg_cdc" plugin_output = "merged" table_match_regex = "<db>\\.partition\\.event_[0-9]{4}-[0-9]{2}-[0-9]{2}" database = "<db>" schema = "public" table = "event" } } sink { Doris { plugin_input = "merged" fenodes = "<fe>:8080" username = "<user>" password = "<password>" database = "<target_db>" table = "${table_name}" sink.enable-2pc = "false" doris.config { format = "json", read_json_by_line = "true" } } } ``` ### Running Command ```shell bin/seatunnel.sh --config job.conf --async curl -s -u <user>:<password> http://127.0.0.1:8080/running-jobs/summary | wc -c ``` ### Error Exception ```log No exception is logged. The endpoint answers 200 with a body of ~28 MB for one such job: its jobName field holds ~16 MB of raw serialized bytes (JSON-escaped). ``` ### Zeta or Flink or Spark Version Zeta (the engine shipped with 3.0.0), single node, REST API v2 on port 8080 ### Java or Scala Version OpenJDK 11.0.33 (Amazon Corretto), Linux x86_64 ### Screenshots _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
