[GitHub] [hudi] vinothchandar commented on pull request #1858: [HUDI-1014] Adding Upgrade and downgrade infra for smooth transitioning from list based rollback to marker based rollback

2020-08-09 Thread GitBox 


vinothchandar commented on pull request #1858:
URL: https://github.com/apache/hudi/pull/1858#issuecomment-671083934


   Pushed some fixes to tests. I think the tests should be testing. lmk if you 
think otherwise. 
   lets iterate quickly and get this in. :) 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #1858: [HUDI-1014] Adding Upgrade and downgrade infra for smooth transitioning from list based rollback to marker based rollback

2020-08-09 Thread GitBox 


vinothchandar commented on pull request #1858:
URL: https://github.com/apache/hudi/pull/1858#issuecomment-671083262


   ```
 // set hoodie.table.version to 0 in hoodie.properties file
   metaClient.getTableConfig().setTableVersion(HoodieTableVersion.ZERO);
   
   if (induceResiduesFromPrevUpgrade) {
 createResidualFile();
   }
   
   // should re-create marker files for 2nd commit since its pending. If 
there was any residues, no upgrade steps should happen except for updating the 
hoodie.table.version
   UpgradeDowngrade.run(metaClient, HoodieTableVersion.ONE, cfg, jsc, null);
   ```
   
   @nsivabalan this won't be a no-op for sure IMO. `setTableVersion()` no 
longer persists the version to dfs. just in memory. but for upgrade it does not 
matter, since default is ZERO. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #1858: [HUDI-1014] Adding Upgrade and downgrade infra for smooth transitioning from list based rollback to marker based rollback

2020-08-09 Thread GitBox 


vinothchandar commented on pull request #1858:
URL: https://github.com/apache/hudi/pull/1858#issuecomment-671080326


   @nsivabalan the concerns you raised around the tests . Are these tests in 
current form not testing anything? is that what you are pointing out? 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #1858: [HUDI-1014] Adding Upgrade and downgrade infra for smooth transitioning from list based rollback to marker based rollback

2020-08-09 Thread GitBox 


vinothchandar commented on pull request #1858:
URL: https://github.com/apache/hudi/pull/1858#issuecomment-671023129


   @nsivabalan updated the PR with changes. 
   
   See `UpgradeDowngrade.java` for the new protocol. I added code to actually 
write the table version into hoodie.properties for new tables. Other than that, 
its mostly code cleanup. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #1858: [HUDI-1014] Adding Upgrade and downgrade infra for smooth transitioning from list based rollback to marker based rollback

2020-08-09 Thread GitBox 


vinothchandar commented on pull request #1858:
URL: https://github.com/apache/hudi/pull/1858#issuecomment-671022017


   New tables have version written now 
   
   ```
   vmacs:HUDIDATA vs$ cat  /tmp/hudi_trips_cow_new/.hoodie/hoodie.properties
   #Properties saved on Sun Aug 09 01:08:18 PDT 2020
   #Sun Aug 09 01:08:18 PDT 2020
   hoodie.table.name=hudi_trips_cow
   hoodie.archivelog.folder=archived
   hoodie.table.type=COPY_ON_WRITE
   hoodie.table.version=1
   hoodie.timeline.layout.version=1
   vmacs:HUDIDATA vs$
   ```
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #1858: [HUDI-1014] Adding Upgrade and downgrade infra for smooth transitioning from list based rollback to marker based rollback

2020-08-09 Thread GitBox 


vinothchandar commented on pull request #1858:
URL: https://github.com/apache/hudi/pull/1858#issuecomment-671021840


   Local test 
   
   ```
   vmacs:HUDIDATA vs$ cat /tmp/hudi_trips_cow/.hoodie/hoodie.properties
   #Properties saved on Sun Aug 09 01:02:02 PDT 2020
   #Sun Aug 09 01:02:02 PDT 2020
   hoodie.table.name=hudi_trips_cow
   hoodie.archivelog.folder=archived
   hoodie.table.type=COPY_ON_WRITE
   hoodie.timeline.layout.version=1
   vmacs:HUDIDATA vs$ cat /tmp/hudi_trips_cow/.hoodie/hoodie.properties
   #Properties saved on Sun Aug 09 01:04:53 PDT 2020
   #Sun Aug 09 01:04:53 PDT 2020
   hoodie.table.name=hudi_trips_cow
   hoodie.archivelog.folder=archived
   hoodie.table.type=COPY_ON_WRITE
   hoodie.table.version=1
   hoodie.timeline.layout.version=1
   
   vmacs:HUDIDATA vs$ ls -R  /tmp/hudi_trips_cow/.hoodie/
   20200809010202.commit20200809010452.commit.requested  archived
   20200809010202.commit.requested  20200809010452.inflight  
hoodie.properties
   20200809010202.inflight  20200809010453.rollback
   20200809010452.commit20200809010453.rollback.inflight
   
   
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #1858: [HUDI-1014] Adding Upgrade and downgrade infra for smooth transitioning from list based rollback to marker based rollback

2020-08-04 Thread GitBox 


vinothchandar commented on pull request #1858:
URL: https://github.com/apache/hudi/pull/1858#issuecomment-668816966


   @nsivabalan I already did that as well. Will review and push some changes to 
this PR 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #1858: [HUDI-1014] Adding Upgrade and downgrade infra for smooth transitioning from list based rollback to marker based rollback

2020-08-03 Thread GitBox 


vinothchandar commented on pull request #1858:
URL: https://github.com/apache/hudi/pull/1858#issuecomment-668403215


   @nsivabalan I rebased this against master



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #1858: [HUDI-1014] Adding Upgrade and downgrade infra for smooth transitioning from list based rollback to marker based rollback

2020-08-03 Thread GitBox 


vinothchandar commented on pull request #1858:
URL: https://github.com/apache/hudi/pull/1858#issuecomment-668286733


   @nsivabalan  can you squash and rebase against master? Also please add the 
great description you have on the PR body into the commit message itself. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org