Hi, pull request to allow asynch retrieval of remote data here: https://github.com/geoserver/geoserver/pull/1091
For the moment I left out the default transformations, but upgraded commons-vfs to 2.0 as I had troubles with a ftp server. Also, vfs 2.0 seems better compatible with our libs, vfs 1.0 was depending on commons http client 2.0, vfs 2.0 on http client 3.1 instead Cheers Andrea On Mon, May 25, 2015 at 8:14 PM, Torben Barsballe < [email protected]> wrote: > Comments inline: > > On Mon, May 25, 2015 at 5:43 AM, Andrea Aime <[email protected] > > wrote: > >> On Thu, May 21, 2015 at 11:01 PM, Torben Barsballe < >> [email protected]> wrote: >> >>> I like the idea of some sort of remote ImportData, but I feel like we >>> would still be better off uploading/configuring it in the createContext >>> step. >>> >>> >>>> So, what about a new type of ImportData, tentatively named FetchData >>>> (feel free to suggest a better name), >>>> >>> >>> How about RemoteData, or OnlineData? >>> >> >> RemoteData, yes, that sounds good. >> >> >>> >>> >>>> No tasks are created during the creation, when run happens, the >>>> importer checks for FetchData, >>>> if available, it will fetch it, create a File or Directory import data, >>>> create the tasks, attach >>>> the transformations >>>> >>> >>> Rather than have this occur in "run", I feel like this step would make >>> more sense in "createContext". I'm not sure about the REST API, but the >>> Importer code does have a createContextAsync method, so this would still >>> support asynchronous behavior. >>> If this option also gets pulled into the importer UI, it would make for >>> a clearer user experience if you could see what is being imported before >>> you import it >>> >> >> Yes, I can go for that, extending the REST api in that direction does not >> look too hard, and makes a lot of sense. >> There is however a catch, in the rest api we need to return ... something. >> Normally it would be a link to the import context just created, but as >> far as I can see createContextAsync does >> return a job id, not a context it. Jobs are not exposed in the REST api, >> and I'm not sure about getting there, >> for example, for asynch run we are still exposing the context, not its >> job. >> >> So here is a counter-proposal in the same spirit as yours. We allow the >> creation of the context, register it >> in the store, but then have a initAsynch(...) that will only do the >> asynchronous initialization. >> This way the REST async creation can still return a link to a Context, >> but one that will be in a new state, >> State.INIT, so that REST clients will get to know the job is still >> initializing, and any call to >> run can be rejected. >> > > I was also running into a similar issue with the createContextAsync (and > runAsync) only returning job ids. Creating and registering the context > first seems like a good solution. One suggestion though is to make > initAsync(...) do almost all of the heavy lifting of setting up the context > (including loading the data) as well as the other asynchronous tasks. > Especially with large databases, creating the context can take almost as > much time as running the import. > So maybe something like registerContext(...) to get a context in > State.INIT, having an Id and nothing else, then initAsync(...) to set up > the context (or maybe instead of an initAsync(...) method simply extend > createContextAsync to also accept an empty context?). > > >> Once init is complete, we can switch to pending and let the REST client >> do its own thing with the tasks >> and eventually run the import. >> >> About the list of "default" transformations in RemoteData, Id' still >> maintain it, if one knows the tasks >> units being imported are uniform, or sort of uniform, it makes sense to >> setup just once the transformations, >> and then eventually work by difference if we know there are a few >> oddballs that are different. >> >> This actually also makes sense for other types of data, say we have for >> example 1000 shapefiles to be >> imported, all needing a reprojection, or something similar. So... what >> about we add a "defaultRasterTransformations" >> and "defaultVectorTransformations" to ImportData (the bast class for all >> ImportData) and just attach >> the tranforms while we create the tasks? >> >> This seems like a good idea. > > Torben > -- == Meet us at the INSPIRE Conference in Lisbon 25-29 May 2015! Visit http://goo.gl/WHKDXT for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via Poggio alle Viti 1187 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003* Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. -------------------------------------------------------
------------------------------------------------------------------------------
_______________________________________________ Geoserver-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geoserver-devel
