Dushan has a very good point (but I think Dushan has or expressed it wrong). It seems that it has been decided to go with Approach 1. i.e. adding the migration capabilities to the existing REST apis of 2.6.0. This means, people using other versions such as 2.1.0, 2.5.0 wont be able to migrate to 3.0 without coming to 2.6.0.
If we had a separate API which can export from any version (lets say from any version from 2.1.0 upwards) and import to 3.0, that is the ideal solution. It is easy to fix issues as well. It is true that this tool will be thrown away after sometime. But, we shouldn't expect all the code we write to be used forever. For me, writing a separate API is easier for us and the users both. On Fri, Sep 21, 2018 at 3:05 AM Dushan Abeyruwan <dus...@wso2.com> wrote: > Hi Samitha, > Are we proceeding with the approach-2? if so, how does such migration > approach benefited for someone who is trying to migrate from 2.5.0 or > before? will there be any intermittent steps or this tool would take care > of such intermittent steps? > > The other question is; its better CLI tool having a verbose option to be > implemented. maybe through (-v) which prints all the necessary details > around the migration state rather someone to enable that via log4j > separately. > > Cheers, > Dushan > > On Wed, Sep 19, 2018 at 9:50 PM, Samitha Chathuranga <sami...@wso2.com> > wrote: > >> Hi all, >> >> Following is an update on the current progress of this task. >> >> Tasks done upto: >> >> 1. 2.6.0 REST API and implementation changes. >> >> - Change GET applications REST API to enable get all apps of a given >> tenant too, instead of all tenants >> - Support cross tenant APIs GET for migration only for super admin >> - Return application keys too (conditionally) to use for migration >> - Support cross tenant application export for migration for super >> admin >> >> 2. Export APIs as a bulk with export-apis command with import-export tool >> >> - All the APIs are exported by single command but implementation is >> as iterative api export, set by set separately (export sets of APIs >> separately, instead of exporting all at once) >> - API export resume capability (error handling) with the support of >> two files migration-apis-export-metadata.yaml and last-succeeded-api.log >> - *--force* flag with *export-apis* command will forcefully export >> all the apis from beginning, disregarding the fact that all/some apis were >> exported successfully by last execution of command. >> - Able to skip exporting particular APIs (if they were failed once >> and not needed to export if execute the command next time without --force >> option) >> >> 3. Fixing the gaps, which had blocked functionality in APIM 3.0.0 >> import-export tool related to API import/export process. >> >> Slight changes on the design of the approach is updated in the same doc >> shared previously. [1] >> Will be updating this thread on the progress and appreciate any concerns. >> >> [1] >> https://docs.google.com/document/d/1N8ZXPf1-dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing >> >> Regards, >> Samitha >> >> On Fri, Sep 7, 2018 at 11:12 AM Samitha Chathuranga <sami...@wso2.com> >> wrote: >> >>> Hi all, >>> >>> This is an update on the up to now state on this task. Had several >>> discussions, design reviews within team and below doc [1] contains the >>> overall design of migration process. And each operation executed at API >>> export/import process (for migration) is detailed under Section 7 (*Using >>> the Import Export CLI tool for migration - User Steps and Steps executed in >>> each command*). >>> >>> Appreciate any comments and suggestions. >>> >>> Meanwhile I have worked on changing required REST APIs for this >>> migration process and are almost done. API/App export related operations >>> from CLI tool is done upto an extent allowing to be finalized based on >>> finalized design. >>> >>> [1] >>> https://docs.google.com/document/d/1N8ZXPf1-dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing >>> >>> Regards, >>> Samitha >>> >>> On Wed, Aug 1, 2018 at 10:31 AM Samitha Chathuranga <sami...@wso2.com> >>> wrote: >>> >>>> Hi Nuwan, >>>> >>>> One main reason is that, if we follow approach 1, migration story >>>> cannot be done in single step. Specially two CLI tools will have to be >>>> used for the two sub steps of exporting and importing. >>>> >>>> And also normal API/Application export flow will change as major >>>> modifications will be required for certain aspects. i.e. >>>> >>>> - bulk export/import >>>> - correlate the difference in the format of archives supported by >>>> 2.6 vs 3.0 >>>> - exporting and importing additional artifacts such as throttle >>>> policies, access tokens, user role permissions >>>> >>>> Importing the 2.6 format API/Applications to 3.0 would require >>>> completely new effort too. So anyway we will have to change the 3.0 import >>>> tool highly. >>>> >>>> And my perception is that normal import export and migration oriented >>>> export-import are tow separate tasks and thought it is better to isolate >>>> the two objectives. (as similarly we do in earlier versions). We will also >>>> have to think about specific permissions required for the migration >>>> oriented export-import in same tool. >>>> >>>> Eventhough we may go with Approach 2, we can reuse the same code bases >>>> when possible. >>>> >>>> But anyway it is a matter of decision, whether we manage the normal >>>> import/export and migration-oriented-import/export separately. Approach 1 >>>> would result with some higher level of import/export while in approach 2, >>>> we could highlight it as a migration. >>>> >>>> Regards, >>>> Samitha >>>> >>>> On Tue, Jul 31, 2018 at 11:08 PM, Nuwan Dias <nuw...@wso2.com> wrote: >>>> >>>>> I would rather go with approach 1. In approach 2 you are suggesting to >>>>> develop a separate web app and a separate tool, why do you think it is >>>>> better than enhancing the current App and CLI? Its obviously going to be a >>>>> lot more work than approach 1, plus at the end of the day we will throw >>>>> away one of those tools because we don't need two tools doing the same >>>>> thing. >>>>> >>>>> On Tue, Jul 31, 2018 at 10:10 PM Samitha Chathuranga <sami...@wso2.com> >>>>> wrote: >>>>> >>>>>> >>>>>> Hi all, >>>>>>> >>>>>>> This is regarding the initial design on possible approaches to >>>>>>> proceed with upgrading APIM 2.6.0(to be released) to 3.0.0. The >>>>>>> migration >>>>>>> process has been already decided to implement through an export/import >>>>>>> approach as a database migration approach won't be realistic as for >>>>>>> older >>>>>>> versions. So the simple generic approach is that each artifact specially >>>>>>> APIs and Applications would be exported from 2.6.0 environment and then >>>>>>> will be imported in to 3.0.0 environment. >>>>>>> >>>>>>> Below I have listed the possible two approaches for both these two >>>>>>> steps (importing & exporting) followed by their pros and cons. >>>>>>> >>>>>>> Exporting : >>>>>>> >>>>>>> Approach 1 : Change currently existing Api Import/export web app and >>>>>>> CLI tool in 2.6 >>>>>>> >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Can use API Import/export Web app and CLI tool and modify >>>>>>> - >>>>>>> >>>>>>> (REST APIs for export api/applications is currently provided >>>>>>> via this WEB app. CLI tool uses this web app) >>>>>>> - >>>>>>> >>>>>>> Required changes: >>>>>>> - >>>>>>> >>>>>>> For app export: Consumer key, Consumer Secret, Token >>>>>>> exporting should be supported in CLI tool >>>>>>> - >>>>>>> >>>>>>> Will have to change admin REST api implementation for above >>>>>>> task and for exporting other artifacts mentioned at the end. >>>>>>> - >>>>>>> >>>>>>> Modify CLI tool to support exporting in bulk. >>>>>>> >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Pros >>>>>>> - >>>>>>> >>>>>>> Will not have to develop/maintain a Web app and CLI tool from >>>>>>> the scratch >>>>>>> - >>>>>>> >>>>>>> Cons >>>>>>> - >>>>>>> >>>>>>> API export flow will change as modifications will be required >>>>>>> - >>>>>>> >>>>>>> Have to manage and consider also about normal export feature >>>>>>> when doing changes for migration-exporting. >>>>>>> - >>>>>>> >>>>>>> Migration story cannot be done in single step. Exporting and >>>>>>> importing will be 2 step process. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Approach 2 : Develop a separate tool without using API Import/export >>>>>>> Web app or CLI tool. >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Create a WEB App introducing new REST APIs required and develop >>>>>>> separate CLI tool. >>>>>>> >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Pros >>>>>>> - >>>>>>> >>>>>>> Migration story can be done in single step. (Exporting and >>>>>>> importing can be done in single command.) >>>>>>> - >>>>>>> >>>>>>> API/Application import-export tool won’t have any effect or >>>>>>> change. >>>>>>> - >>>>>>> >>>>>>> Don’t have to manage 2 separate use cases in single tool. >>>>>>> - >>>>>>> >>>>>>> Cons >>>>>>> - >>>>>>> >>>>>>> Will have to develop/maintain a Web app and/or CLI tool from >>>>>>> the scratch >>>>>>> >>>>>>> >>>>>>> >>>>>>> Importing : >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Has to create separate REST APIs for importing 2.6-exported >>>>>>> APIs/Applications/other artifacts as 3.0.0 expecting archives in 3.0 >>>>>>> specific format. >>>>>>> >>>>>>> >>>>>>> Approach 1 >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Change/improve API Import/export CLI tool in 3.0 to support >>>>>>> importing 2.6-exported API/application archives. >>>>>>> >>>>>>> Approach 2 >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Develop separate CLI tool from the scratch which will be used >>>>>>> for exporting from 2.6 also >>>>>>> >>>>>>> >>>>>>> >>>>>>> Other artifacts required to be exported/imported (other than >>>>>>> APIs/Applications) >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Throttling Policies >>>>>>> - >>>>>>> >>>>>>> Custom API Lifecycles >>>>>>> - >>>>>>> >>>>>>> Workflows >>>>>>> - >>>>>>> >>>>>>> User Role Permissions >>>>>>> - >>>>>>> >>>>>>> Custom sequences (at least for reference -> has to decide >>>>>>> regarding this) >>>>>>> >>>>>>> >>>>>>> >>>>>>> My perception is that proceeding with Approach 2 under both >>>>>>> exporting and importing would be better considering the pros and cons of >>>>>>> each. >>>>>>> >>>>>>> Highly appreciate your thoughts on this. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Samitha >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Samitha Chathuranga* >>>>>>> *Senior Software Engineer*, *WSO2 Inc.* >>>>>>> lean.enterprise.middleware >>>>>>> Mobile: +94715123761 >>>>>>> >>>>>>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Nuwan Dias >>>>> >>>>> Director - WSO2, Inc. http://wso2.com >>>>> email : nuw...@wso2.com >>>>> Phone : +94 777 775 729 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Samitha Chathuranga* >>>> *Senior Software Engineer*, *WSO2 Inc.* >>>> lean.enterprise.middleware >>>> Mobile: +94715123761 >>>> >>>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>>> >>> >>> >>> -- >>> *Samitha Chathuranga* >>> *Senior Software Engineer*, *WSO2 Inc.* >>> lean.enterprise.middleware >>> Mobile: +94715123761 >>> >>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>> >> >> >> -- >> *Samitha Chathuranga* >> *Senior Software Engineer*, *WSO2 Inc.* >> lean.enterprise.middleware >> Mobile: +94715123761 >> >> [image: http://wso2.com/signature] <http://wso2.com/signature> >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Dushan Abeyruwan > Architect(Associate Director)-Delivery/Support > Technical Support,MV > PMC Member Apache Synpase > WSO2 Inc. http://wso2.com/ > Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>* > LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan > <https://www.linkedin.com/in/dushanabeyruwan>* > Mobile:(001)408-987-1348 > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > -- *Amila Maharachchi* Director of Engineering WSO2, Inc.; http://wso2.com Blog: http://maharachchi.blogspot.com Mobile: +94719371446
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
