[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=228716&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228716
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 16/Apr/19 22:08
Start Date: 16/Apr/19 22:08
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on pull request #8289: 
[BEAM-7064] Reject BigQuery data with sub-millisecond precision, instead of 
losing data
URL: https://github.com/apache/beam/pull/8289
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 228716)
Time Spent: 4h 10m  (was: 4h)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=228547&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228547
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 16/Apr/19 16:35
Start Date: 16/Apr/19 16:35
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-483737187
 
 
   run java precommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 228547)
Time Spent: 4h  (was: 3h 50m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=228073&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228073
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 16/Apr/19 01:15
Start Date: 16/Apr/19 01:15
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #8289: [BEAM-7064] Reject 
BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-483474028
 
 
   +1 the flag should opt in not opt out.
   
   On Mon, Apr 15, 2019 at 4:09 PM Kenn Knowles 
   wrote:
   
   > run sql postcommit
   >
   > —
   > You are receiving this because your review was requested.
   > Reply to this email directly, view it on GitHub
   > , or mute
   > the thread
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 228073)
Time Spent: 3h 50m  (was: 3h 40m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=228036&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228036
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 15/Apr/19 23:09
Start Date: 15/Apr/19 23:09
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-483451678
 
 
   run sql postcommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 228036)
Time Spent: 3h 40m  (was: 3.5h)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=228010&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228010
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 15/Apr/19 22:35
Start Date: 15/Apr/19 22:35
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-483443814
 
 
   run java precommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 228010)
Time Spent: 3.5h  (was: 3h 20m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=228003&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228003
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 15/Apr/19 22:16
Start Date: 15/Apr/19 22:16
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-483438908
 
 
   Because of how the TableRow to Beam Row conversion is currently expressed a 
bit differently (as a map of typename to parser), it will be a moderate 
refactor to share these conversion options. I don't want to confuse this review 
with that, but will start work on it since we don't want to lose data there 
either.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 228003)
Time Spent: 3h 20m  (was: 3h 10m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=228002&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228002
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 15/Apr/19 22:13
Start Date: 15/Apr/19 22:13
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-483437987
 
 
   Updated to have a flag that can be set in table properties. I thought this 
code was shared with BigQuery schema conversions, but it does not appear to be.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 228002)
Time Spent: 3h 10m  (was: 3h)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=227882&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227882
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 15/Apr/19 18:49
Start Date: 15/Apr/19 18:49
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-483372775
 
 
   I feel strongly that we should not truncate by default.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 227882)
Time Spent: 3h  (was: 2h 50m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=227865&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227865
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 15/Apr/19 17:56
Start Date: 15/Apr/19 17:56
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #8289: [BEAM-7064] Reject 
BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-483354915
 
 
   With a flag, which behavior should be the default?
   
   Aggressive plan is to fail users pipelines so make exception approach as the 
default. 
   
   Or we can be more cautious to allow truncating timestamp by default.
   
   It's more like a short term pain (possibly breaking change to some users) vs 
long term pain (transparent precision truncation that is hard to detect for 
both existing and new users). 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 227865)
Time Spent: 2h 50m  (was: 2h 40m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=227844&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227844
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 15/Apr/19 17:16
Start Date: 15/Apr/19 17:16
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-483340610
 
 
   I'm cool with an opt-in flag.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 227844)
Time Spent: 2h 40m  (was: 2.5h)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=227805&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227805
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 15/Apr/19 16:15
Start Date: 15/Apr/19 16:15
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #8289: [BEAM-7064] Reject 
BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-483319207
 
 
   So I'm a bit worried about this change: there may be users who are ok with 
losing timestamp precision (e.g. for log events, they may perceive losing nano 
precision on event timestamps as a noop). Maybe for now we should allow them to 
override this behavior with a flag?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 227805)
Time Spent: 2.5h  (was: 2h 20m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=227020&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227020
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 23:36
Start Date: 12/Apr/19 23:36
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-482753276
 
 
   Not user friendly at all! An attempt to limit data loss. The real solution 
will be nano precision, a logical type at the Beam schema level. We will have 
to reimplement date time functions.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 227020)
Time Spent: 2h 20m  (was: 2h 10m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226957&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226957
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 21:58
Start Date: 12/Apr/19 21:58
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #8289: [BEAM-7064] Reject 
BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-482736066
 
 
   LGTM
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226957)
Time Spent: 2h 10m  (was: 2h)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226868&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226868
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 19:22
Start Date: 12/Apr/19 19:22
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #8289: [BEAM-7064] Reject 
BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-482693285
 
 
   Run Java PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226868)
Time Spent: 2h  (was: 1h 50m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226867&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226867
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 19:21
Start Date: 12/Apr/19 19:21
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on pull request #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#discussion_r275031566
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
 ##
 @@ -73,8 +69,19 @@ public static Object convertAvroFormat(Field beamField, 
Object value) {
   default:
 throw new RuntimeException("Does not support converting unknown type 
value");
 }
+  }
 
-return ret;
+  private static ReadableInstant safeToMillis(Object value) {
+long subMilliPrecision = ((long) value) % 1000;
 
 Review comment:
   I was also wondering where long is generated. If BQ timestamp converts to 
long which contains micros, will that lose precision?
   
   I did some math:
   
   BigQuery doc says TIMESTAMP type range is `0001-01-01 00:00:00 to -12-31 
23:59:59.99 UTC`
   
   so it should ranges from `-6.127488×10¹⁶ to 2.4976512×10¹⁷` in micros, 
calculated by `([0001-01-01 | -12-31] - 1970-01-01) * 12 * 30 * 24 *3600 * 
1000 * 1000`, which falls into INT64's range. 
   
   So no precision loss.
   
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226867)
Time Spent: 1h 50m  (was: 1h 40m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226863&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226863
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 19:13
Start Date: 12/Apr/19 19:13
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on pull request #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#discussion_r275031566
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
 ##
 @@ -73,8 +69,19 @@ public static Object convertAvroFormat(Field beamField, 
Object value) {
   default:
 throw new RuntimeException("Does not support converting unknown type 
value");
 }
+  }
 
-return ret;
+  private static ReadableInstant safeToMillis(Object value) {
+long subMilliPrecision = ((long) value) % 1000;
 
 Review comment:
   I was also wondering where long is generated. If BQ timestamp converts to 
long which contains micros, will that lose precision?
   
   I did some math:
   
   BigQuery doc says TIMESTAMP type range is `0001-01-01 00:00:00 to -12-31 
23:59:59.99 UTC`
   
   so it should ranges from `-6.127488×10¹⁶ to 2.4976512×10¹⁷` in micros, 
calculated by `([0001-01-01 | -12-31] - 1970) * 12 * 30 * 24 *3600 * 1000 * 
1000`, which falls into INT64's range. 
   
   So no precision loss.
   
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226863)
Time Spent: 1h 40m  (was: 1.5h)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226862&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226862
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 19:13
Start Date: 12/Apr/19 19:13
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on pull request #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#discussion_r275031566
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
 ##
 @@ -73,8 +69,19 @@ public static Object convertAvroFormat(Field beamField, 
Object value) {
   default:
 throw new RuntimeException("Does not support converting unknown type 
value");
 }
+  }
 
-return ret;
+  private static ReadableInstant safeToMillis(Object value) {
+long subMilliPrecision = ((long) value) % 1000;
 
 Review comment:
   I was also wondering where long is generated. If BQ timestamp converts to 
long which contains micros, will that lose precision?
   
   I did some math:
   
   BigQuery doc says TIMESTAMP type range is `0001-01-01 00:00:00 to -12-31 
23:59:59.99 UTC`
   
   so it should ranges from `-6.127488×10¹⁶ to 2.4976512×10¹⁷` in micros, 
calculated by `((0001-01-01 / -12-31) - 1970) * 12 * 30 * 24 *3600 * 1000 * 
1000`, which falls into INT64's range. 
   
   So no precision loss.
   
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226862)
Time Spent: 1.5h  (was: 1h 20m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226861&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226861
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 19:12
Start Date: 12/Apr/19 19:12
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on pull request #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#discussion_r275031566
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
 ##
 @@ -73,8 +69,19 @@ public static Object convertAvroFormat(Field beamField, 
Object value) {
   default:
 throw new RuntimeException("Does not support converting unknown type 
value");
 }
+  }
 
-return ret;
+  private static ReadableInstant safeToMillis(Object value) {
+long subMilliPrecision = ((long) value) % 1000;
 
 Review comment:
   I was also wondering where long is generated. If BQ timestamp converts to 
long which contains micros, will that lose precision?
   
   I did some math:
   
   BigQuery doc says TIMESTAMP type range is `0001-01-01 00:00:00 to -12-31 
23:59:59.99 UTC`
   
   so it should ranges from `-6.127488×10¹⁶ to 2.4976512×10¹⁷` in micros, 
calculated by `((0001-01-01 / -12-31) -/+ 1970) * 12 * 30 * 24 *3600 * 1000 
* 1000`, which falls into INT64's range. 
   
   So no precision loss.
   
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226861)
Time Spent: 1h 20m  (was: 1h 10m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226857
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 19:04
Start Date: 12/Apr/19 19:04
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on pull request #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#discussion_r275029235
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
 ##
 @@ -73,8 +69,19 @@ public static Object convertAvroFormat(Field beamField, 
Object value) {
   default:
 throw new RuntimeException("Does not support converting unknown type 
value");
 }
+  }
 
-return ret;
+  private static ReadableInstant safeToMillis(Object value) {
+long subMilliPrecision = ((long) value) % 1000;
 
 Review comment:
   Yes. I was asking because input is an object.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226857)
Time Spent: 1h 10m  (was: 1h)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226855&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226855
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 19:00
Start Date: 12/Apr/19 19:00
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on pull request #8289: 
[BEAM-7064] Reject BigQuery data with sub-millisecond precision, instead of 
losing data
URL: https://github.com/apache/beam/pull/8289#discussion_r275027986
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
 ##
 @@ -73,8 +69,19 @@ public static Object convertAvroFormat(Field beamField, 
Object value) {
   default:
 throw new RuntimeException("Does not support converting unknown type 
value");
 }
+  }
 
-return ret;
+  private static ReadableInstant safeToMillis(Object value) {
+long subMilliPrecision = ((long) value) % 1000;
 
 Review comment:
   It is because this `AvroUtils` is part of the BigQuery connector, not 
general. And we know BigQuery is always micros. Is that your question?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226855)
Time Spent: 1h  (was: 50m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226838&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226838
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 18:31
Start Date: 12/Apr/19 18:31
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #8289: [BEAM-7064] Reject 
BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-482677983
 
 
   Run SQL PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226838)
Time Spent: 50m  (was: 40m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226837&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226837
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 18:29
Start Date: 12/Apr/19 18:29
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on pull request #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#discussion_r275017781
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
 ##
 @@ -73,8 +69,19 @@ public static Object convertAvroFormat(Field beamField, 
Object value) {
   default:
 throw new RuntimeException("Does not support converting unknown type 
value");
 }
+  }
 
-return ret;
+  private static ReadableInstant safeToMillis(Object value) {
+long subMilliPrecision = ((long) value) % 1000;
 
 Review comment:
   How do we know that that the value returned is not millis but micros?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226837)
Time Spent: 40m  (was: 0.5h)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226836&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226836
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 18:28
Start Date: 12/Apr/19 18:28
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on pull request #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#discussion_r275017781
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
 ##
 @@ -73,8 +69,19 @@ public static Object convertAvroFormat(Field beamField, 
Object value) {
   default:
 throw new RuntimeException("Does not support converting unknown type 
value");
 }
+  }
 
-return ret;
+  private static ReadableInstant safeToMillis(Object value) {
+long subMilliPrecision = ((long) value) % 1000;
 
 Review comment:
   How do we know that that the value returned is not millis but micro?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226836)
Time Spent: 0.5h  (was: 20m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226762&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226762
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 16:30
Start Date: 12/Apr/19 16:30
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #8289: [BEAM-7064] 
Reject BigQuery data with sub-millisecond precision, instead of losing data
URL: https://github.com/apache/beam/pull/8289#issuecomment-482639099
 
 
   R: @amaliujia too
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 226762)
Time Spent: 20m  (was: 10m)

> Conversion of timestamp from BigQuery row to Beam row loses precision
> -
>
> Key: BEAM-7064
> URL: https://issues.apache.org/jira/browse/BEAM-7064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently, the utilities to convert from BigQuery row to Beam row simply 
> truncate timestamps at millisecond precision. This is unacceptable. Instead, 
> an error should be raised indicating that it is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7064) Conversion of timestamp from BigQuery row to Beam row loses precision

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7064?focusedWorklogId=226470&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226470
 ]

ASF GitHub Bot logged work on BEAM-7064:


Author: ASF GitHub Bot
Created on: 12/Apr/19 03:51
Start Date: 12/Apr/19 03:51
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on pull request #8289: 
[BEAM-7064] Reject BigQuery data with sub-millisecond precision, instead of 
losing data
URL: https://github.com/apache/beam/pull/8289
 
 
   Previously, reading from BQ and converting to Beam rows - either via SQL or 
just using schema-driven transforms - would cause timestamps to be truncated at 
milliseconds. We don't support higher precision, but this rejects pipelines 
that would otherwise lose data.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [x] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
 | --- | --- | ---