[
https://issues.apache.org/jira/browse/GOBBLIN-1150?focusedWorklogId=434125&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434125
]
ASF GitHub Bot logged work on GOBBLIN-1150:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 17/May/20 00:41
Start Date: 17/May/20 00:41
Worklog Time Spent: 10m
Work Description: sv2000 commented on a change in pull request #2988:
URL: https://github.com/apache/incubator-gobblin/pull/2988#discussion_r426203594
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStore.java
##########
@@ -65,31 +65,49 @@
private static final String CREATE_TABLE_STATEMENT =
"CREATE TABLE IF NOT EXISTS %s (spec_uri VARCHAR(128) NOT NULL, tag
VARCHAR(128) NOT NULL, spec LONGBLOB, PRIMARY KEY (spec_uri))";
+ private static final String CREATE_TABLE_STATEMENT_V2 =
Review comment:
Isn't this schema backwards compatible with the V1 schema i.e. you are
simply adding new columns? We can retain the blob format for the "spec" column
as before and define a new column "specV2" which stores the spec in JSON
format. If so, do we need to support two different versions of the table?
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStore.java
##########
@@ -65,31 +65,49 @@
private static final String CREATE_TABLE_STATEMENT =
"CREATE TABLE IF NOT EXISTS %s (spec_uri VARCHAR(128) NOT NULL, tag
VARCHAR(128) NOT NULL, spec LONGBLOB, PRIMARY KEY (spec_uri))";
+ private static final String CREATE_TABLE_STATEMENT_V2 =
+ "CREATE TABLE IF NOT EXISTS %s (spec_uri VARCHAR(128) NOT NULL,
flow_group VARCHAR(128), flow_name VARCHAR(128), "
Review comment:
I think the schema can be enhanced with a few additional fields such as
requesterId, a boolean field "isRunImmediately" to indicate if the scheduled
flow is also a run immediately flow, a futuristic field "owningGroup" for flows
which are owned by a group, "timezone" to indicate the timezone the cron
schedule applies to. Also, does user_to_proxy need to be a top level field? It
applies only when the orchestrator is Azkaban and can probably be queried from
the JSON spec.
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStore.java
##########
@@ -140,11 +172,22 @@ public boolean deleteSpec(Spec spec) throws IOException {
@Override
public boolean deleteSpec(URI specUri) throws IOException {
try (Connection connection = this.dataSource.getConnection();
- PreparedStatement statement =
connection.prepareStatement(String.format(DELETE_STATEMENT, this.tableName))) {
+ PreparedStatement statement =
connection.prepareStatement(String.format(DELETE_STATEMENT, this.tableName));
+ PreparedStatement statementV2 =
connection.prepareStatement(String.format(DELETE_STATEMENT, this.tableNameV2)))
{
+
statement.setString(1, specUri.toString());
- int result = statement.executeUpdate();
+ statementV2.setString(1, specUri.toString());
+
+ int resultV2 = statementV2.executeUpdate();
+ if (this.writeToOldTable) {
+ int result = statement.executeUpdate();
+ if (resultV2 != result) {
Review comment:
Should we throw an exception and fail the deleteSpec, instead of logging
and moving on?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 434125)
Remaining Estimate: 0h
Time Spent: 10m
> change spec store table to use json type
> ----------------------------------------
>
> Key: GOBBLIN-1150
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1150
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Arjun Singh Bora
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
