[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-05-05 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=430989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430989
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 06/May/20 01:58
Start Date: 06/May/20 01:58
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on pull request #11034:
URL: https://github.com/apache/beam/pull/11034#issuecomment-624403737


   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 430989)
Time Spent: 4h 50m  (was: 4h 40m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-05 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=398521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-398521
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 05/Mar/20 17:21
Start Date: 05/Mar/20 17:21
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-595347570
 
 
   IT's not just performance critical code. We wanted these transforms to be
   usable by code that doesn't understand the logical type.
   
   For example: you should be able to write a SQL expression over a type that
   contains a OneOf. SQL doesn't understand the OneOf logical type of course,
   however the base type is a Row with nullable fields, Since today getValue
   returns the base type, you should be able to query it. Similar story with
   Enumerations - you should be able to treat them as integers in a SQL
   statement or in one of the schema transforms. (note: I"m not sure if this
   has been tested with SQL, so it may not work today).
   
   We need to figure out a good API that allows these transformation to use
   the base type by default but _also_ allows you to write a combiner that
   accepts the appropriate value type.
   
   Reuven
   
   On Thu, Mar 5, 2020 at 9:12 AM Gleb Kanterov 
   wrote:
   
   > @reuvenlax  is there an option to have
   > alternative to attachValues, but for getting values to be used in
   > performance-critical code?
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 398521)
Time Spent: 4h 40m  (was: 4.5h)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-05 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=398511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-398511
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 05/Mar/20 17:12
Start Date: 05/Mar/20 17:12
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-595343397
 
 
   @reuvenlax is there an option to have alternative to attachValues, but for 
getting values to be used in performance-critical code? 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 398511)
Time Spent: 4.5h  (was: 4h 20m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-05 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=398480=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-398480
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 05/Mar/20 16:33
Start Date: 05/Mar/20 16:33
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-595322444
 
 
   @kanterov this is because (as I mentioned before) Row.getValue today always 
returns the base type (in this case int), because that's how we store it. This 
has the side effect of passing the base value into your combiner, which is 
currently expecting the value type.
   
   This inconsistency keeps causing strange effects like this. I think we need 
getValue to return the same types passed into addValues (the value type). I'm 
trying to think of a good way to do this without breaking other semantics and 
impacting performance.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 398480)
Time Spent: 4h 20m  (was: 4h 10m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-05 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=398464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-398464
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 05/Mar/20 16:09
Start Date: 05/Mar/20 16:09
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-595310241
 
 
   @kanterov it's failing differently now - no longer in. SelectHelpers.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 398464)
Time Spent: 4h 10m  (was: 4h)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-05 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=398453=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-398453
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 05/Mar/20 15:53
Start Date: 05/Mar/20 15:53
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-595302297
 
 
   @reuvenlax I've rebased on the latest master that includes 
https://github.com/apache/beam/pull/11025. It didn't seem to help, because the 
test is failing with the same exception:
   
   
https://builds.apache.org/job/beam_PreCommit_Java_Commit/10259/testReport/junit/org.apache.beam.sdk.schemas.transforms/GroupTest/testAggregateLogicalType/
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 398453)
Time Spent: 4h  (was: 3h 50m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-04 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=397772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397772
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 04/Mar/20 18:58
Start Date: 04/Mar/20 18:58
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-594753144
 
 
   BTW I think I need to prioritize fixing the inconsistency between getValues 
and addValues, as this is not the first time it's caused a bug.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 397772)
Time Spent: 3h 50m  (was: 3h 40m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-04 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=397679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397679
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 04/Mar/20 16:41
Start Date: 04/Mar/20 16:41
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-594651288
 
 
   @kanterov ok, I see the bug now. the problem is because we have an 
unfortunate asymmetry between Row.Builder.addValues and Row.getValues. 
getValues returns the base type for logical types (because that's how it's 
stored, but addValues expects to get the input type.
   
   I'm still thinking of the best way to resolve this inconsistency without 
breaking other use cases. However a simple fix for now is to fix SelectHelpers 
to use attachValues instead of addValues. Coincidentally that change is already 
made in https://github.com/apache/beam/pull/11025. Can you check to see whether 
that PR fixes the problem you're seeing?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 397679)
Time Spent: 3h 40m  (was: 3.5h)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-04 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=397678=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397678
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 04/Mar/20 16:35
Start Date: 04/Mar/20 16:35
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-594648165
 
 
   @reuvenlax I cleaned the code and left only test that reproduces this issue.
   
   The reason for that is because it uses `RowCoderGenerator` that would use a 
`Coder` instead of `Coder` (see `RowCoderGenerator.java:374`). 
While it doesn't make any difference for binary representation, SDK code fails 
because of that. In the stack trace we can see how Combine should get 
`InputT=EnumerationType$Value`, but it gets `BaseT=Integer`.
   
   ```
   Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to 
org.apache.beam.sdk.schemas.logicaltypes.EnumerationType$Value
at 
org.apache.beam.sdk.schemas.logicaltypes.EnumerationType.toBaseType(EnumerationType.java:37)
at 
org.apache.beam.sdk.values.Row$Builder.verifyLogicalType(Row.java:659)
at org.apache.beam.sdk.values.Row$Builder.verify(Row.java:652)
at org.apache.beam.sdk.values.Row$Builder.verify(Row.java:635)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:840)
at 
org.apache.beam.sdk.schemas.utils.SelectHelpers.selectRow(SelectHelpers.java:194)
at 
org.apache.beam.sdk.schemas.transforms.SchemaAggregateFn$Inner$ExtractSingleFieldFunction.apply(SchemaAggregateFn.java:241)
at 
org.apache.beam.sdk.schemas.transforms.SchemaAggregateFn$Inner$ExtractSingleFieldFunction.apply(SchemaAggregateFn.java:231)
at 
org.apache.beam.sdk.transforms.CombineFns$ComposedCombineFn.addInput(CombineFns.java:377)
at 
org.apache.beam.sdk.schemas.transforms.SchemaAggregateFn$Inner.addInput(SchemaAggregateFn.java:283)
at 
org.apache.beam.sdk.schemas.transforms.SchemaAggregateFn$Inner.addInput(SchemaAggregateFn.java:55)
at 
org.apache.beam.sdk.transforms.Combine$CombineFn.apply(Combine.java:446)
at 
org.apache.beam.sdk.transforms.Combine$GroupedValues$1.processElement(Combine.java:2171)
   ```
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 397678)
Time Spent: 3.5h  (was: 3h 20m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-04 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=397669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397669
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 04/Mar/20 15:58
Start Date: 04/Mar/20 15:58
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-594625264
 
 
   The whole point of Logical types is that the representation should be that 
of the base type (the schema returned by getBaseType). This means that the 
logical type can be processed by code that doesn't know about InputT, and also 
makes it possible to use it cross language. 
   
   Where in Group do we need Coder? I feel like I'm still missing 
something.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 397669)
Time Spent: 3h 20m  (was: 3h 10m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-04 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=397404=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397404
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 04/Mar/20 09:31
Start Date: 04/Mar/20 09:31
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-594414157
 
 
   @reuvenlax because `RowCoderGenerator` uses `Coder`, not 
`Coder`. We know how to produce `Coder`, and we can make 
`Coder` out of it using `LogicalType`. So the test fails because Row 
builder expected `EnumerationType.Value` but got `String`. What is special 
about Group is that it constructs coders for keys and values.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 397404)
Time Spent: 3h 10m  (was: 3h)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-04 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=397399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397399
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 04/Mar/20 09:29
Start Date: 04/Mar/20 09:29
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-594414157
 
 
   @reuvenlax because `RowCoderGenerator` uses `Coder`, not 
`Coder`. We know how to produce `Coder`, and we can make 
`Coder` out of it using `LogicalType`. So the test fails because Row 
builder expected `EnumerationType.Value` but got `String`. What is special 
about Group is that it constructs 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 397399)
Time Spent: 3h  (was: 2h 50m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=397315=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397315
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 04/Mar/20 07:35
Start Date: 04/Mar/20 07:35
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-594370177
 
 
   I'm not sure I understand the problem or the solution. Why don't logical 
types work in Group? Why does creating a new LogicalTypeCoder solve this?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 397315)
Time Spent: 2h 50m  (was: 2h 40m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=397120=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397120
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 23:08
Start Date: 03/Mar/20 23:08
Worklog Time Spent: 10m 
  Work Description: kanterov commented on pull request #11034: [BEAM-9424] 
Allow grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#discussion_r387349711
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
 ##
 @@ -401,6 +404,20 @@ private static StackManipulation 
getCoder(Schema.FieldType fieldType) {
 }
   }
 
+  private static StackManipulation logicalTypeCoder(
+  Schema.LogicalType logicalType, StackManipulation baseCoder) {
+throw new UnsupportedOperationException("not implemented");
 
 Review comment:
   This section is left unimplemented, and that's why the test is failing
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 397120)
Time Spent: 2h 40m  (was: 2.5h)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=397119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397119
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 23:07
Start Date: 03/Mar/20 23:07
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11034: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034#issuecomment-594217964
 
 
   @reuvenlax it looks like there is a couple of places where LogicalType don't 
work since we introduced to/fromBaseType functions. Looking into the code we 
would need to support it in RowCoderGenerator that doesn't look straightforward 
because LogicalType doesn't have a static factory we can call.
   
   Do you have thoughts on how we can fix that?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 397119)
Time Spent: 2.5h  (was: 2h 20m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=397116=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397116
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 23:03
Start Date: 03/Mar/20 23:03
Worklog Time Spent: 10m 
  Work Description: kanterov commented on pull request #11034: [BEAM-9424] 
Allow grouping by LogicalType
URL: https://github.com/apache/beam/pull/11034
 
 
   Add LogicalTypeCoder.
   
   Support LogicalTypes in Group transform and SchemaCoder generation.
   
   Related to: https://github.com/apache/beam/pull/11015
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 

[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396857
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 17:03
Start Date: 03/Mar/20 17:03
Worklog Time Spent: 10m 
  Work Description: kanterov commented on pull request #11015: [BEAM-9424] 
Allow grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396857)
Time Spent: 2h 10m  (was: 2h)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396856
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 17:03
Start Date: 03/Mar/20 17:03
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11015: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015#issuecomment-594058397
 
 
   LGTM
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396856)
Time Spent: 2h  (was: 1h 50m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396834
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 16:09
Start Date: 03/Mar/20 16:09
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11015: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015#issuecomment-594030815
 
 
   retest this please
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396834)
Time Spent: 1h 50m  (was: 1h 40m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396790
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 14:40
Start Date: 03/Mar/20 14:40
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11015: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015#issuecomment-593984596
 
 
   retest this please
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396790)
Time Spent: 1h 40m  (was: 1.5h)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396788=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396788
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 14:39
Start Date: 03/Mar/20 14:39
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11015: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015#issuecomment-593984171
 
 
   Run SQL PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396788)
Time Spent: 1.5h  (was: 1h 20m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396787=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396787
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 14:38
Start Date: 03/Mar/20 14:38
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11015: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015#issuecomment-593983612
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396787)
Time Spent: 1h 20m  (was: 1h 10m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396780=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396780
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 14:30
Start Date: 03/Mar/20 14:30
Worklog Time Spent: 10m 
  Work Description: fediazgon commented on pull request #11015: [BEAM-9424] 
Allow grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015
 
 
   `GROUP BY` statement fails if field is `LogicalType`. E.g., when converting 
a BQ schema with either `TIME`, `DATE` or `DATETIME` to a Beam schema and 
grouping by one or more of these fields. 
   
   This PR add supports for grouping by LogicalType.
   
   This fixes https://issues.apache.org/jira/browse/BEAM-9424.
   
   **Please** add a meaningful description for your change here
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [x] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 

[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396779
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 14:30
Start Date: 03/Mar/20 14:30
Worklog Time Spent: 10m 
  Work Description: kanterov commented on pull request #11015: [BEAM-9424] 
Allow grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396779)
Time Spent: 1h  (was: 50m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396778
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 14:30
Start Date: 03/Mar/20 14:30
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11015: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015#issuecomment-593979628
 
 
   Jenkins doesn't build
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396778)
Time Spent: 50m  (was: 40m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396776
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 14:27
Start Date: 03/Mar/20 14:27
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11015: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015#issuecomment-593978095
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396776)
Time Spent: 40m  (was: 0.5h)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Assignee: fdiazgon
>Priority: Minor
>  Labels: sql
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396704
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 08:54
Start Date: 03/Mar/20 08:54
Worklog Time Spent: 10m 
  Work Description: kanterov commented on pull request #11015: [BEAM-9424] 
Allow grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015#discussion_r386873928
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamComplexTypeTest.java
 ##
 @@ -373,6 +376,45 @@ public void testNullInnerRow() {
 pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
   }
 
+  @Test
+  public void testLogicalTypes() {
+Schema beamSchemaWithLogicalTypes =
+BigQueryUtils.fromTableSchema(
 
 Review comment:
   Is it possible to use `DummySqlTimeType`  and `DummySqlDateType` as in other 
tests? I find dependency on gcp-io not necessary
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396704)
Time Spent: 0.5h  (was: 20m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Priority: Blocker
>  Labels: sql
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-03 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396702=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396702
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 03/Mar/20 08:52
Start Date: 03/Mar/20 08:52
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #11015: [BEAM-9424] Allow 
grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015#issuecomment-593836769
 
 
   R: @kanterov 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396702)
Time Spent: 20m  (was: 10m)

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Priority: Blocker
>  Labels: sql
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Work logged] (BEAM-9424) Grouping By LogicalTypes is not supported

2020-03-02 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/BEAM-9424?focusedWorklogId=396171=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396171
 ]

ASF GitHub Bot logged work on BEAM-9424:


Author: ASF GitHub Bot
Created on: 02/Mar/20 16:02
Start Date: 02/Mar/20 16:02
Worklog Time Spent: 10m 
  Work Description: fediazgon commented on pull request #11015: [BEAM-9424] 
Allow grouping by LogicalType
URL: https://github.com/apache/beam/pull/11015
 
 
   `GROUP BY` statement fails if field is `LogicalType`. E.g., when converting 
a BQ schema with either `TIME`, `DATE` or `DATETIME` to a Beam schema and 
grouping by one or more of these fields. 
   
   This PR add supports for grouping by LogicalType.
   
   This fixes https://issues.apache.org/jira/browse/BEAM-9424.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 396171)
Remaining Estimate: 0h
Time Spent: 10m

> Grouping By LogicalTypes is not supported
> -
>
> Key: BEAM-9424
> URL: https://issues.apache.org/jira/browse/BEAM-9424
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.19.0
>Reporter: fdiazgon
>Priority: Blocker
>  Labels: sql
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Creating a schema from a BQ schema that has either TIME, DATE or DATETIME 
> columns, and grouping by one of these fields throws NullPointerException.
> {code:java}
> Pipeline pipeline = Pipeline.create();
> Schema beamSchemaWithLogicalTypes =
> BigQueryUtils.fromTableSchema(
> new TableSchema()
> .setFields(
> Arrays.asList(
> new TableFieldSchema().setName("fTime").setType("TIME"),
> new TableFieldSchema().setName("fDate").setType("DATE"),
> new 
> TableFieldSchema().setName("fDatetime").setType("DATETIME";
> Row row =
> Row.withSchema(beamSchemaWithLogicalTypes)
> .addValues(
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02"),
> DateTime.parse("2020-02-02T00:00:00"))
> .build();
> PCollection outputRow =
> pipeline
> .apply(Create.of(row))
> .setRowSchema(beamSchemaWithLogicalTypes)
> .apply(
> SqlTransform.query(
> "SELECT fTime, fDate, fDatetime FROM PCOLLECTION GROUP BY 
> fTime, fDate, fDatetime"));
> pipeline.run();
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)