[ https://issues.apache.org/jira/browse/BEAM-9010?focusedWorklogId=362835&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362835 ]
ASF GitHub Bot logged work on BEAM-9010: ---------------------------------------- Author: ASF GitHub Bot Created on: 24/Dec/19 01:19 Start Date: 24/Dec/19 01:19 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10444: [BEAM-9010] Proper TableRow size calculation via TableRowJsonCoder URL: https://github.com/apache/beam/pull/10444#issuecomment-568619521 Run BeamSQL_ZetaSQL PreCommit ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 362835) Time Spent: 2h 10m (was: 2h) > BigQuery TableRow's size is toString().length() ? > ------------------------------------------------- > > Key: BEAM-9010 > URL: https://issues.apache.org/jira/browse/BEAM-9010 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow > Reporter: Tomo Suzuki > Assignee: Tomo Suzuki > Priority: Minor > Attachments: TableRowJsonCoder_behavior_remains_same.png > > Time Spent: 2h 10m > Remaining Estimate: 0h > > The following tests failed when I tried to upgrade google-http-client 1.34.0 > from 1.28.0: > {noformat} > org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOReadTest.testEstimatedSizeWithoutStreamingBuffer > org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOReadTest.testEstimatedSizeWithStreamingBuffer > org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtilTest.testInsertAll > {noformat} > [https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink] > h3. Reason of the test failures > [org.apache.beam.sdk.io.gcp.testing.TableContainer|https://github.com/apache/beam/blob/6fa94c9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/TableContainer.java#L43] > and > [org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl|https://github.com/apache/beam/blob/c2f0d28/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L758] > rely on {{TableRow.toString().length()}} to calculate the size. Example: > {code:java} > dataSize += row.toString().length(); > if (dataSize >= maxRowBatchSize > || rows.size() >= maxRowsPerBatch > || i == rowsToPublish.size() - 1) { > {code} > However, with [google-http-client's > PR#589|https://github.com/googleapis/google-http-java-client/pull/589/files#diff-914cd7ff18143b3d2398149e1cfb4f45R218], > the GenericData.toString output has changed since v1.29.0. > In old google-http-client 1.28.0, an example row's toString returned: > {noformat} > {f=[{v=foo}, {v=1234}]} > {noformat} > In new google-http-client 1.29.0 and higher, the same row's toString returns: > {noformat} > GenericData{classInfo=[f], {f=[GenericData{classInfo=[v], {v=foo}}, > GenericData{classInfo=[v], {v=1234}}]}} > {noformat} > h1. Question: > Is this right thing to rely on {{toString().length()}} in the BigQuery > classes? -- This message was sent by Atlassian Jira (v8.3.4#803005)