[
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226861&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226861
]
ASF GitHub Bot logged work on BEAM-7064:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Apr/19 19:12
Start Date: 12/Apr/19 19:12
Worklog Time Spent: 10m
Work Description: amaliujia commented on pull request #8289: [BEAM-7064]
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#discussion_r275031566
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
##########
@@ -73,8 +69,19 @@ public static Object convertAvroFormat(Field beamField,
Object value) {
default:
throw new RuntimeException("Does not support converting unknown type
value");
}
+ }
- return ret;
+ private static ReadableInstant safeToMillis(Object value) {
+ long subMilliPrecision = ((long) value) % 1000;
Review comment:
I was also wondering where long is generated. If BQ timestamp converts to
long which contains micros, will that lose precision?
I did some math:
BigQuery doc says TIMESTAMP type range is `0001-01-01 00:00:00 to 9999-12-31
23:59:59.999999 UTC`
so it should ranges from `-6.127488×10¹⁶ to 2.4976512×10¹⁷` in micros,
calculated by `((0001-01-01 / 9999-12-31) -/+ 1970) * 12 * 30 * 24 *3600 * 1000
* 1000`, which falls into INT64's range.
So no precision loss.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 226861)
Time Spent: 1h 20m (was: 1h 10m)
> Conversion of timestamp from BigQuery row to Beam row loses precision
> ---------------------------------------------------------------------
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Kenneth Knowles
> Assignee: Kenneth Knowles
> Priority: Critical
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply
> truncate timestamps at millisecond precision. This is unacceptable. Instead,
> an error should be raised indicating that it is not supported.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
