Hi, This is an update on the status of this migration implementation.
REST API and related Impl/DAO layer changes are done as discussed previously and below is a summary on that. [1]. I'm creating the WUM update for 2.6.0 for these changes (PRs : [2] & [3] ) and will be released within few days. *Description on the WUM update* Cross tenant APIs/Apps export support through REST APIs is required for the APIM 2.6.0/2.x to APIM 3.0.0 migration. This was not available previously. Changes done are described below. *For exporting APIs* Get APIs list - Change Publisher REST API to enable Cross tenant API list retrieval - Add new parameter to the /apis GET to specify tenant - Use conditions to return APIs of another tenant - System parameter set at server startup : migrationMode - Allowed only If the user is a super tenant admin and server is started with -DmigrationMode=true property set. Export APIs - Import-export REST API (web app) changes (via starting tenantFlow) to enable Cross tenant API Exporting. - This will be allowed only to super-tenant admin user and only if the ‘migrationMode’ system property is set. *For exporting Applications* Get list of Applications For Applications export - Change Admin REST API - /applications GET to get list of all the apps for the given tenant/cross tenants too. (previously, only getting the applications accessible to a single user was possible ) - Add new parameter to the /applications GET to specify tenant domain - Validate special permissions to allow only the super tenant admin users, only when the server is started with migrationMode=true property, to get cross tenant application list For exporting Applications - Change Admin REST API - /export/applications GET to export Apps of the given tenant/cross tenants too. - Return application keys too (conditionally) to use for migration - Validate special permissions to allow only the super tenant admin users, only when the server is started with migrationMode=true property, for cross tenant apps export ------ Tested these changes though multiple approaches and no issues found. Created applications in store via different tenant users, tenant admins and tested the admin REST API whether cross tenant application list retrieval and App exporting is working as expected, adhering to the migrationMode system property and super admin user role without violating the expected policies. Created APIs in publisher via different tenant users, tenant admins and tested the Publisher REST API and Import-export REST API, whether the cross tenant api list retrieval and api exporting is working as expected , adhering to the migrationMode property and super admin user role without violating the expected policies. And also executed product integration tests on a pack in which migrationMode system property is enabled too. [1] https://github.com/wso2/product-apim/issues/3901 [2] https://github.com/wso2-support/carbon-apimgt/pull/1138 [3] https://github.com/wso2-support/product-apim/pull/325 Regards, Samitha On Fri, Sep 21, 2018 at 11:11 AM Samitha Chathuranga <sami...@wso2.com> wrote: > Hi, > > @Amila Maha Arachchi <ami...@wso2.com> > >> For me, writing a separate API is easier for us and the users both. >> > As Nuwan has described above, supporting to be migrated from any version > (2.1.0/2.2.0/2.5.0..) can be a complex and error prone task and so not > healthy for us. Mainly it would need adding new REST APIs or changing > existing APIs to all these older versions and that is cumbersome. > > @Dushan Abeyruwan <dus...@wso2.com> > >> The other question is; its better CLI tool having a verbose option to be >> implemented. maybe through (-v) which prints all the necessary details >> around the migration state rather someone to enable that via log4j >> separately. >> > Verbose mode is already configured in import-export tool and so available > for all the commands. So it is upto us to drop some logs from non-verbose > mode and include only in verbose mode. My feeling is that it is better to > log as much information in a non verbose mode too, because that would help > us dig through errors by default. Migration is not something done always > hence the logs are not a thing which would continuously pile up. And if we > limit some logs to verbose mode, then we may require to request our > customers to enable verbose mode and do migration again and send the logs. > > Regards, > Samitha > > On Fri, Sep 21, 2018 at 10:02 AM Nuwan Dias <nuw...@wso2.com> wrote: > >> Let's take this incrementally. We need to do lots of changes on every >> version to support this kind of migration. In addition to that the changes >> between the versions are not identical. So every migration needs duplicated >> work and testing to be carried out. Plus whatever changes we do to each of >> those old version have to shipped as WUM updates and the clients needs to >> get those WUM updates for proceeding. We all know that allowing too many >> options ends up with too many problems, in the end of the day ending up >> with too many frustrated customers. >> >> The migration to API Manager 3.0 is going to be the most sophisticated >> API Manager migration we have done so far. Given our history of migration >> experiences trying to do too many thing first up is not a good idea. So I >> really want us to focus on 2.6 to 3.0 migration and make it really smooth >> and robust before attempting anything else. >> >> Thanks, >> NuwanD. >> >> On Fri, Sep 21, 2018 at 9:51 AM Amila Maha Arachchi <ami...@wso2.com> >> wrote: >> >>> Dushan has a very good point (but I think Dushan has or expressed it >>> wrong). It seems that it has been decided to go with Approach 1. i.e. >>> adding the migration capabilities to the existing REST apis of 2.6.0. This >>> means, people using other versions such as 2.1.0, 2.5.0 wont be able to >>> migrate to 3.0 without coming to 2.6.0. >>> >>> If we had a separate API which can export from any version (lets say >>> from any version from 2.1.0 upwards) and import to 3.0, that is the ideal >>> solution. It is easy to fix issues as well. It is true that this tool will >>> be thrown away after sometime. But, we shouldn't expect all the code we >>> write to be used forever. >>> >>> For me, writing a separate API is easier for us and the users both. >>> >>> On Fri, Sep 21, 2018 at 3:05 AM Dushan Abeyruwan <dus...@wso2.com> >>> wrote: >>> >>>> Hi Samitha, >>>> Are we proceeding with the approach-2? if so, how does such migration >>>> approach benefited for someone who is trying to migrate from 2.5.0 or >>>> before? will there be any intermittent steps or this tool would take care >>>> of such intermittent steps? >>>> >>>> The other question is; its better CLI tool having a verbose option to >>>> be implemented. maybe through (-v) which prints all the necessary details >>>> around the migration state rather someone to enable that via log4j >>>> separately. >>>> >>>> Cheers, >>>> Dushan >>>> >>>> On Wed, Sep 19, 2018 at 9:50 PM, Samitha Chathuranga <sami...@wso2.com> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Following is an update on the current progress of this task. >>>>> >>>>> Tasks done upto: >>>>> >>>>> 1. 2.6.0 REST API and implementation changes. >>>>> >>>>> - Change GET applications REST API to enable get all apps of a >>>>> given tenant too, instead of all tenants >>>>> - Support cross tenant APIs GET for migration only for super admin >>>>> - Return application keys too (conditionally) to use for migration >>>>> - Support cross tenant application export for migration for super >>>>> admin >>>>> >>>>> 2. Export APIs as a bulk with export-apis command with import-export >>>>> tool >>>>> >>>>> - All the APIs are exported by single command but implementation >>>>> is as iterative api export, set by set separately (export sets of APIs >>>>> separately, instead of exporting all at once) >>>>> - API export resume capability (error handling) with the support >>>>> of two files migration-apis-export-metadata.yaml and >>>>> last-succeeded-api.log >>>>> - *--force* flag with *export-apis* command will forcefully export >>>>> all the apis from beginning, disregarding the fact that all/some apis >>>>> were >>>>> exported successfully by last execution of command. >>>>> - Able to skip exporting particular APIs (if they were failed once >>>>> and not needed to export if execute the command next time without >>>>> --force >>>>> option) >>>>> >>>>> 3. Fixing the gaps, which had blocked functionality in APIM 3.0.0 >>>>> import-export tool related to API import/export process. >>>>> >>>>> Slight changes on the design of the approach is updated in the same >>>>> doc shared previously. [1] >>>>> Will be updating this thread on the progress and appreciate any >>>>> concerns. >>>>> >>>>> [1] >>>>> https://docs.google.com/document/d/1N8ZXPf1-dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing >>>>> >>>>> Regards, >>>>> Samitha >>>>> >>>>> On Fri, Sep 7, 2018 at 11:12 AM Samitha Chathuranga <sami...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> This is an update on the up to now state on this task. Had several >>>>>> discussions, design reviews within team and below doc [1] contains the >>>>>> overall design of migration process. And each operation executed at API >>>>>> export/import process (for migration) is detailed under Section 7 (*Using >>>>>> the Import Export CLI tool for migration - User Steps and Steps executed >>>>>> in >>>>>> each command*). >>>>>> >>>>>> Appreciate any comments and suggestions. >>>>>> >>>>>> Meanwhile I have worked on changing required REST APIs for this >>>>>> migration process and are almost done. API/App export related operations >>>>>> from CLI tool is done upto an extent allowing to be finalized based on >>>>>> finalized design. >>>>>> >>>>>> [1] >>>>>> https://docs.google.com/document/d/1N8ZXPf1-dKpYf09w7AZ9M6RHXnsU2Orht37N1yETbnE/edit?usp=sharing >>>>>> >>>>>> Regards, >>>>>> Samitha >>>>>> >>>>>> On Wed, Aug 1, 2018 at 10:31 AM Samitha Chathuranga <sami...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Nuwan, >>>>>>> >>>>>>> One main reason is that, if we follow approach 1, migration story >>>>>>> cannot be done in single step. Specially two CLI tools will have to >>>>>>> be used for the two sub steps of exporting and importing. >>>>>>> >>>>>>> And also normal API/Application export flow will change as major >>>>>>> modifications will be required for certain aspects. i.e. >>>>>>> >>>>>>> - bulk export/import >>>>>>> - correlate the difference in the format of archives supported >>>>>>> by 2.6 vs 3.0 >>>>>>> - exporting and importing additional artifacts such as throttle >>>>>>> policies, access tokens, user role permissions >>>>>>> >>>>>>> Importing the 2.6 format API/Applications to 3.0 would require >>>>>>> completely new effort too. So anyway we will have to change the 3.0 >>>>>>> import >>>>>>> tool highly. >>>>>>> >>>>>>> And my perception is that normal import export and migration >>>>>>> oriented export-import are tow separate tasks and thought it is better >>>>>>> to >>>>>>> isolate the two objectives. (as similarly we do in earlier versions). We >>>>>>> will also have to think about specific permissions required for the >>>>>>> migration oriented export-import in same tool. >>>>>>> >>>>>>> Eventhough we may go with Approach 2, we can reuse the same code >>>>>>> bases when possible. >>>>>>> >>>>>>> But anyway it is a matter of decision, whether we manage the normal >>>>>>> import/export and migration-oriented-import/export separately. Approach >>>>>>> 1 >>>>>>> would result with some higher level of import/export while in approach >>>>>>> 2, >>>>>>> we could highlight it as a migration. >>>>>>> >>>>>>> Regards, >>>>>>> Samitha >>>>>>> >>>>>>> On Tue, Jul 31, 2018 at 11:08 PM, Nuwan Dias <nuw...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I would rather go with approach 1. In approach 2 you are suggesting >>>>>>>> to develop a separate web app and a separate tool, why do you think it >>>>>>>> is >>>>>>>> better than enhancing the current App and CLI? Its obviously going to >>>>>>>> be a >>>>>>>> lot more work than approach 1, plus at the end of the day we will throw >>>>>>>> away one of those tools because we don't need two tools doing the same >>>>>>>> thing. >>>>>>>> >>>>>>>> On Tue, Jul 31, 2018 at 10:10 PM Samitha Chathuranga < >>>>>>>> sami...@wso2.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> This is regarding the initial design on possible approaches to >>>>>>>>>> proceed with upgrading APIM 2.6.0(to be released) to 3.0.0. The >>>>>>>>>> migration >>>>>>>>>> process has been already decided to implement through an >>>>>>>>>> export/import >>>>>>>>>> approach as a database migration approach won't be realistic as for >>>>>>>>>> older >>>>>>>>>> versions. So the simple generic approach is that each artifact >>>>>>>>>> specially >>>>>>>>>> APIs and Applications would be exported from 2.6.0 environment and >>>>>>>>>> then >>>>>>>>>> will be imported in to 3.0.0 environment. >>>>>>>>>> >>>>>>>>>> Below I have listed the possible two approaches for both these >>>>>>>>>> two steps (importing & exporting) followed by their pros and cons. >>>>>>>>>> >>>>>>>>>> Exporting : >>>>>>>>>> >>>>>>>>>> Approach 1 : Change currently existing Api Import/export web app >>>>>>>>>> and CLI tool in 2.6 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Can use API Import/export Web app and CLI tool and modify >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> (REST APIs for export api/applications is currently >>>>>>>>>> provided via this WEB app. CLI tool uses this web app) >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Required changes: >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> For app export: Consumer key, Consumer Secret, Token >>>>>>>>>> exporting should be supported in CLI tool >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Will have to change admin REST api implementation for >>>>>>>>>> above task and for exporting other artifacts mentioned at the >>>>>>>>>> end. >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Modify CLI tool to support exporting in bulk. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Pros >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Will not have to develop/maintain a Web app and CLI tool >>>>>>>>>> from the scratch >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Cons >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> API export flow will change as modifications will be >>>>>>>>>> required >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Have to manage and consider also about normal export >>>>>>>>>> feature when doing changes for migration-exporting. >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Migration story cannot be done in single step. Exporting >>>>>>>>>> and importing will be 2 step process. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Approach 2 : Develop a separate tool without using API >>>>>>>>>> Import/export Web app or CLI tool. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Create a WEB App introducing new REST APIs required and >>>>>>>>>> develop separate CLI tool. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Pros >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Migration story can be done in single step. (Exporting and >>>>>>>>>> importing can be done in single command.) >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> API/Application import-export tool won’t have any effect >>>>>>>>>> or change. >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Don’t have to manage 2 separate use cases in single tool. >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Cons >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Will have to develop/maintain a Web app and/or CLI tool >>>>>>>>>> from the scratch >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Importing : >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Has to create separate REST APIs for importing 2.6-exported >>>>>>>>>> APIs/Applications/other artifacts as 3.0.0 expecting archives in >>>>>>>>>> 3.0 >>>>>>>>>> specific format. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Approach 1 >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Change/improve API Import/export CLI tool in 3.0 to support >>>>>>>>>> importing 2.6-exported API/application archives. >>>>>>>>>> >>>>>>>>>> Approach 2 >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Develop separate CLI tool from the scratch which will be used >>>>>>>>>> for exporting from 2.6 also >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Other artifacts required to be exported/imported (other than >>>>>>>>>> APIs/Applications) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Throttling Policies >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Custom API Lifecycles >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Workflows >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> User Role Permissions >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> Custom sequences (at least for reference -> has to decide >>>>>>>>>> regarding this) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> My perception is that proceeding with Approach 2 under both >>>>>>>>>> exporting and importing would be better considering the pros and >>>>>>>>>> cons of >>>>>>>>>> each. >>>>>>>>>> >>>>>>>>>> Highly appreciate your thoughts on this. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Samitha >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Samitha Chathuranga* >>>>>>>>>> *Senior Software Engineer*, *WSO2 Inc.* >>>>>>>>>> lean.enterprise.middleware >>>>>>>>>> Mobile: +94715123761 >>>>>>>>>> >>>>>>>>>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Nuwan Dias >>>>>>>> >>>>>>>> Director - WSO2, Inc. http://wso2.com >>>>>>>> email : nuw...@wso2.com >>>>>>>> Phone : +94 777 775 729 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Samitha Chathuranga* >>>>>>> *Senior Software Engineer*, *WSO2 Inc.* >>>>>>> lean.enterprise.middleware >>>>>>> Mobile: +94715123761 >>>>>>> >>>>>>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Samitha Chathuranga* >>>>>> *Senior Software Engineer*, *WSO2 Inc.* >>>>>> lean.enterprise.middleware >>>>>> Mobile: +94715123761 >>>>>> >>>>>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Samitha Chathuranga* >>>>> *Senior Software Engineer*, *WSO2 Inc.* >>>>> lean.enterprise.middleware >>>>> Mobile: +94715123761 >>>>> >>>>> [image: http://wso2.com/signature] <http://wso2.com/signature> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Dushan Abeyruwan >>>> Architect(Associate Director)-Delivery/Support >>>> Technical Support,MV >>>> PMC Member Apache Synpase >>>> WSO2 Inc. http://wso2.com/ >>>> Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>* >>>> LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan >>>> <https://www.linkedin.com/in/dushanabeyruwan>* >>>> Mobile:(001)408-987-1348 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>> >>> >>> -- >>> *Amila Maharachchi* >>> Director of Engineering >>> WSO2, Inc.; http://wso2.com >>> >>> Blog: http://maharachchi.blogspot.com >>> Mobile: +94719371446 >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >> >> >> -- >> *Nuwan Dias* | Director | WSO2 Inc. >> (m) +94 777 775 729 | (e) nuw...@wso2.com >> [image: Signature.jpg] >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> > > > -- > *Samitha Chathuranga* > *Senior Software Engineer*, *WSO2 Inc.* > lean.enterprise.middleware > Mobile: +94715123761 > > [image: http://wso2.com/signature] <http://wso2.com/signature> > -- *Samitha Chathuranga* *Senior Software Engineer*, *WSO2 Inc.* lean.enterprise.middleware Mobile: +94715123761 [image: http://wso2.com/signature] <http://wso2.com/signature>
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
