alexey-bass opened a new issue, #13273:
URL: https://github.com/apache/seatunnel/issues/13273

   ### Search before asking
   
   - [x] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
 and found no similar issues.
   
   ### What happened
   
   With a job whose serialized `JobImmutableInformation` is larger than about 
16 MB (in our case a Postgres-CDC job with a `table-pattern` matching ~730 
tables), `GET /running-jobs/summary` returns a response of tens of megabytes: 
`jobName` contains ~16 MB of raw serialized bytes. The web UI's Overview and 
Jobs pages load this endpoint and hang in the browser. `/running-jobs` and 
`/job-info/<id>` return correct data, and the job itself runs normally.
   
   **Cause**
   
   `JobInfoService.decodeJobBasicInfo` reads the fields directly from the 
`Data` buffer:
   
   ```java
   // IdentifiedDataSerializable header: factoryId + classId.
   in.readInt();
   in.readInt();
   in.readLong(); // jobId
   String jobName = in.readString();
   in.readBoolean(); // isStartWithSavePoint
   long createTime = in.readLong();
   ```
   
   Hazelcast's `DataSerializableSerializer` writes a 1-byte header (identified 
/ versioned flags) before `factoryId` and `classId`. The fast path does not 
skip it, so every following read is shifted by one byte. `readString()` then 
takes its length from the last byte of `jobId` followed by the first three 
bytes of the real length, i.e. `(jobId & 0xFF) << 24` for any normal job name. 
All job ids in our cluster end in `0x01`, so the length is 16,777,216.
   
   - Serialized job smaller than that: `readString` hits EOF, the `catch 
(Throwable)` fallback does the full deserialization and the result is correct. 
This hides the bug for normal jobs (and means the fast path never actually 
takes effect).
   - Serialized job larger than that: `readString` succeeds and returns 16 MB 
of the buffer as `jobName`; `createTime` is garbage too.
   
   **Expected**
   
   `jobName` and `createTime` of the job; a response of a few hundred bytes per 
job.
   
   **How to reproduce**
   
   Submit any job whose `JobImmutableInformation` exceeds 16 MB (for example a 
CDC source with several hundred wide tables), then call `GET 
/running-jobs/summary` or open the UI Overview page.
   
   **Suggested fix**
   
   Skip the header byte before the two ints (`in.readByte()`), and additionally 
validate the decoded name (for example, fall back to full deserialization when 
the length is implausible). A unit test with a `JobImmutableInformation` larger 
than 16 MB would cover it.
   
   ### SeaTunnel Version
   
   3.0.0 (the method is unchanged on `dev`)
   
   ### SeaTunnel Config
   
   ```conf
   # shape of the job; names and hosts replaced
   env {
     job.name = "cdc_example"
     job.mode = "STREAMING"
     parallelism = 1
     checkpoint.interval = 30000
   }
   source {
     Postgres-CDC {
       plugin_output = "pg_cdc"
       url = "jdbc:postgresql://<host>:5432/<db>"
       username = "<user>"
       password = "<password>"
       database-names = ["<db>"]
       # matches ~730 daily tables with ~54 columns each
       table-pattern = "<db>\\.(partition\\.event_([0-9]{4}-[0-9]{2}-[0-9]{2}))"
       database-pattern = "<db>"
       decoding.plugin.name = "pgoutput"
       slot.name = "<slot>"
       startup.mode = "latest"
     }
   }
   transform {
     TableMerge {
       plugin_input = "pg_cdc"
       plugin_output = "merged"
       table_match_regex = "<db>\\.partition\\.event_[0-9]{4}-[0-9]{2}-[0-9]{2}"
       database = "<db>"
       schema = "public"
       table = "event"
     }
   }
   sink {
     Doris {
       plugin_input = "merged"
       fenodes = "<fe>:8080"
       username = "<user>"
       password = "<password>"
       database = "<target_db>"
       table = "${table_name}"
       sink.enable-2pc = "false"
       doris.config { format = "json", read_json_by_line = "true" }
     }
   }
   ```
   
   ### Running Command
   
   ```shell
   bin/seatunnel.sh --config job.conf --async
   curl -s -u <user>:<password> http://127.0.0.1:8080/running-jobs/summary | wc 
-c
   ```
   
   ### Error Exception
   
   ```log
   No exception is logged. The endpoint answers 200 with a body of ~28 MB for 
one such job: its jobName field holds ~16 MB of raw serialized bytes 
(JSON-escaped).
   ```
   
   ### Zeta or Flink or Spark Version
   
   Zeta (the engine shipped with 3.0.0), single node, REST API v2 on port 8080
   
   ### Java or Scala Version
   
   OpenJDK 11.0.33 (Amazon Corretto), Linux x86_64
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to