Re: [Qgis-developer] Report 10 - QGIS Resources Sharing Tools

[Matthias Kuhn]

Hi Akbar,

Do you really need a flag? I think you can just check if the scheme is
specified in the URI. If it's missing, assume the same protocol.

Cheers

On 08/01/2016 11:35 AM, Akbar Gumbira wrote:
> Hi Matthias,
> 
> I think I missed your point before. I think I will add a flag in the
> metadata that users need to put, to state whether the repository is
> using the same protocol as the metadata or not. This way it could be
> more portable to use the repository in different protocol. For example
> users can clone a repository from git and use that repository in the
> local without having to change the metadata file itself.
> 
> Cheers
> 
> 
> 
> On Mon, Aug 1, 2016 at 11:13 AM, Akbar Gumbira  > wrote:
> 
> Hi Matthias,
>  
> 
> I guess support for relative URLs should be trivial and make the
> repositories more portable. And as far as I can see it should be
> possible to have relative and absolute paths side-by-side, no?
> 
> If it's a matter of portability, users can still put the metadata
> inside the repository itself (if they put the repository in a
> directory based e.g git/local file system). It's just that they need
> to explicitly put the metadata URI in the settings when adding a
> repository. 
> 
> If users want to make a zip collections, they need to put the
> metadata somewhere else outside the zip. I agree with Ale though
> that most users probably prefer this, they can just put a zip and
> the metadata file in dropbox or google drive for example.
> 
> Cheers
> 
> On Mon, Aug 1, 2016 at 10:56 AM, Matthias Kuhn  > wrote:
> 
> Hi
> 
> > In general, I feels that this is a better approach also in the sense
> > that "explicit is better than implicit", we provide explicit URIs
> > instead of delegate guessing them to the software layer.
> 
> I guess support for relative URLs should be trivial and make the
> repositories more portable. And as far as I can see it should be
> possible to have relative and absolute paths side-by-side, no?
> 
> Regards
> Matthias
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> 
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> 
> 
> 
> 
> -- 
> 
> *---*
> 
> *Akbar Gumbira *
> *www.akbargumbira.com *
> 
> 
> 
> 
> -- 
> 
> *---*
> 
> *Akbar Gumbira *
> *www.akbargumbira.com *
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Report 10 - QGIS Resources Sharing Tools

[Akbar Gumbira]

Hi Matthias,

I think I missed your point before. I think I will add a flag in the
metadata that users need to put, to state whether the repository is using
the same protocol as the metadata or not. This way it could be more
portable to use the repository in different protocol. For example users can
clone a repository from git and use that repository in the local without
having to change the metadata file itself.

Cheers



On Mon, Aug 1, 2016 at 11:13 AM, Akbar Gumbira 
wrote:

> Hi Matthias,
>
>
>> I guess support for relative URLs should be trivial and make the
>> repositories more portable. And as far as I can see it should be
>> possible to have relative and absolute paths side-by-side, no?
>
> If it's a matter of portability, users can still put the metadata inside
> the repository itself (if they put the repository in a directory based e.g
> git/local file system). It's just that they need to explicitly put the
> metadata URI in the settings when adding a repository.
>
> If users want to make a zip collections, they need to put the metadata
> somewhere else outside the zip. I agree with Ale though that most users
> probably prefer this, they can just put a zip and the metadata file in
> dropbox or google drive for example.
>
> Cheers
>
> On Mon, Aug 1, 2016 at 10:56 AM, Matthias Kuhn 
> wrote:
>
>> Hi
>>
>> > In general, I feels that this is a better approach also in the sense
>> > that "explicit is better than implicit", we provide explicit URIs
>> > instead of delegate guessing them to the software layer.
>>
>> I guess support for relative URLs should be trivial and make the
>> repositories more portable. And as far as I can see it should be
>> possible to have relative and absolute paths side-by-side, no?
>>
>> Regards
>> Matthias
>> ___
>> Qgis-developer mailing list
>> Qgis-developer@lists.osgeo.org
>> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
>
>
> --
>
> *---*
> *Akbar Gumbira *
> *www.akbargumbira.com *
>



-- 

*---*
*Akbar Gumbira *
*www.akbargumbira.com *
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Report 10 - QGIS Resources Sharing Tools

[Akbar Gumbira]

Hi Matthias,


> I guess support for relative URLs should be trivial and make the
> repositories more portable. And as far as I can see it should be
> possible to have relative and absolute paths side-by-side, no?

If it's a matter of portability, users can still put the metadata inside
the repository itself (if they put the repository in a directory based e.g
git/local file system). It's just that they need to explicitly put the
metadata URI in the settings when adding a repository.

If users want to make a zip collections, they need to put the metadata
somewhere else outside the zip. I agree with Ale though that most users
probably prefer this, they can just put a zip and the metadata file in
dropbox or google drive for example.

Cheers

On Mon, Aug 1, 2016 at 10:56 AM, Matthias Kuhn  wrote:

> Hi
>
> > In general, I feels that this is a better approach also in the sense
> > that "explicit is better than implicit", we provide explicit URIs
> > instead of delegate guessing them to the software layer.
>
> I guess support for relative URLs should be trivial and make the
> repositories more portable. And as far as I can see it should be
> possible to have relative and absolute paths side-by-side, no?
>
> Regards
> Matthias
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer




-- 

*---*
*Akbar Gumbira *
*www.akbargumbira.com *
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Report 10 - QGIS Resources Sharing Tools

[Matthias Kuhn]

Hi

> In general, I feels that this is a better approach also in the sense
> that "explicit is better than implicit", we provide explicit URIs
> instead of delegate guessing them to the software layer.

I guess support for relative URLs should be trivial and make the
repositories more portable. And as far as I can see it should be
possible to have relative and absolute paths side-by-side, no?

Regards
Matthias
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Report 10 - QGIS Resources Sharing Tools

[Akbar Gumbira]

Ok thanks Ale. I'll focus on separating metadata handler and repository
handler this week.

Cheers

On Mon, Aug 1, 2016 at 10:24 AM, Alessandro Pasotti 
wrote:

> On Mon, Aug 1, 2016 at 10:17 AM, Akbar Gumbira 
> wrote:
>
>> Hi Ale
>>
>>
>>> Thanks for your report Akbar!
>>>
>>
>>
>>> I'm sorry if you've not taken into account my repeated calls to keep
>>> authentication and proxy support in the primary goals. As I explained you
>>> this is very important for some companies and organizations that needs
>>> authenticated endpoints in their networks.
>>>
>>
>>
>>> If this plugin will not use this feature, the a.m. organization will not
>>> be able to use it (and they will not invest any time or resources in its
>>> maintainance or future development), they will just build their own sharing
>>> solution.
>>>
>>
>>
>>> For this reason I've been pushing since the very beginning to use
>>> QgsNetworkAccessManager for all network calls, in order to be able to use
>>> QGIS proxy settings and authentication plugins.
>>
>>
>> Agreed. In the network class, I used QgsNetworkAccessManager. The tricky
>> part is the Dulwich's porcelain used to interact with git repositories. For
>> this, I think we need to do it manually through git configuration to use
>> proxy.
>>
>
>
> In this case we will loose authentication, but I guess it's not a great
> issue: I don't think that git will be the preferred sharing protocol in
> case of authenticated repos.
>
>
>
>>
>> The algorithm could be:
>>> # Browsing:
>>> - for each the repo url:
>>>- if it does not point to a metadata.ini file
>>>   - get the metadata address from the URL
>>>- if we have an handler to retrieve the metadata.ini
>>>   - retrieve the metadata.ini
>>>- else
>>>   - error
>>> #Installing
>>> - if the metadata does not contain a download URL for the collection
>>> - build the collection download URL from the repo URL and protocol
>>> and collection name
>>> - if we have an handler to retrieve the collection
>>> - download and install
>>> - else
>>> - error
>>
>>
>> I was thinking to separate between Metadata Handler and Repository
>> Handler as they don't have any correlation. For example, the metadata could
>> be in an ftp and the repository could be in git. Doing if else (treating
>> the only one git case: that the repository URL is a git repository and
>> building a metadata URL from that URL is not that worth it). So when users
>> add a repository, they add the metadata url. What do you think?
>>
>>
>
> It looks good to me, browsing metadata and installing a collection are two
> separate steps (in time and scope) and it's worth splitting the URLs into
> different protocol, if needed.
>
> Of course, for most repos, the handler and protocol will just be the same
> for both operations.
>
> In general, I feels that this is a better approach also in the sense that
> "explicit is better than implicit", we provide explicit URIs instead of
> delegate guessing them to the software layer.
>
> Cheers
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>



-- 

*---*
*Akbar Gumbira *
*www.akbargumbira.com *
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Report 10 - QGIS Resources Sharing Tools

[Alessandro Pasotti]

On Mon, Aug 1, 2016 at 10:17 AM, Akbar Gumbira 
wrote:

> Hi Ale
>
>
>> Thanks for your report Akbar!
>>
>
>
>> I'm sorry if you've not taken into account my repeated calls to keep
>> authentication and proxy support in the primary goals. As I explained you
>> this is very important for some companies and organizations that needs
>> authenticated endpoints in their networks.
>>
>
>
>> If this plugin will not use this feature, the a.m. organization will not
>> be able to use it (and they will not invest any time or resources in its
>> maintainance or future development), they will just build their own sharing
>> solution.
>>
>
>
>> For this reason I've been pushing since the very beginning to use
>> QgsNetworkAccessManager for all network calls, in order to be able to use
>> QGIS proxy settings and authentication plugins.
>
>
> Agreed. In the network class, I used QgsNetworkAccessManager. The tricky
> part is the Dulwich's porcelain used to interact with git repositories. For
> this, I think we need to do it manually through git configuration to use
> proxy.
>


In this case we will loose authentication, but I guess it's not a great
issue: I don't think that git will be the preferred sharing protocol in
case of authenticated repos.



>
> The algorithm could be:
>> # Browsing:
>> - for each the repo url:
>>- if it does not point to a metadata.ini file
>>   - get the metadata address from the URL
>>- if we have an handler to retrieve the metadata.ini
>>   - retrieve the metadata.ini
>>- else
>>   - error
>> #Installing
>> - if the metadata does not contain a download URL for the collection
>> - build the collection download URL from the repo URL and protocol
>> and collection name
>> - if we have an handler to retrieve the collection
>> - download and install
>> - else
>> - error
>
>
> I was thinking to separate between Metadata Handler and Repository Handler
> as they don't have any correlation. For example, the metadata could be in
> an ftp and the repository could be in git. Doing if else (treating the only
> one git case: that the repository URL is a git repository and building a
> metadata URL from that URL is not that worth it). So when users add a
> repository, they add the metadata url. What do you think?
>
>

It looks good to me, browsing metadata and installing a collection are two
separate steps (in time and scope) and it's worth splitting the URLs into
different protocol, if needed.

Of course, for most repos, the handler and protocol will just be the same
for both operations.

In general, I feels that this is a better approach also in the sense that
"explicit is better than implicit", we provide explicit URIs instead of
delegate guessing them to the software layer.

Cheers

-- 
Alessandro Pasotti
w3:   www.itopen.it
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Report 10 - QGIS Resources Sharing Tools

[Akbar Gumbira]

Hi Ale


> Thanks for your report Akbar!
>


> I'm sorry if you've not taken into account my repeated calls to keep
> authentication and proxy support in the primary goals. As I explained you
> this is very important for some companies and organizations that needs
> authenticated endpoints in their networks.
>


> If this plugin will not use this feature, the a.m. organization will not
> be able to use it (and they will not invest any time or resources in its
> maintainance or future development), they will just build their own sharing
> solution.
>


> For this reason I've been pushing since the very beginning to use
> QgsNetworkAccessManager for all network calls, in order to be able to use
> QGIS proxy settings and authentication plugins.


Agreed. In the network class, I used QgsNetworkAccessManager. The tricky
part is the Dulwich's porcelain used to interact with git repositories. For
this, I think we need to do it manually through git configuration to use
proxy.

The algorithm could be:
> # Browsing:
> - for each the repo url:
>- if it does not point to a metadata.ini file
>   - get the metadata address from the URL
>- if we have an handler to retrieve the metadata.ini
>   - retrieve the metadata.ini
>- else
>   - error
> #Installing
> - if the metadata does not contain a download URL for the collection
> - build the collection download URL from the repo URL and protocol and
> collection name
> - if we have an handler to retrieve the collection
> - download and install
> - else
> - error


I was thinking to separate between Metadata Handler and Repository Handler
as they don't have any correlation. For example, the metadata could be in
an ftp and the repository could be in git. Doing if else (treating the only
one git case: that the repository URL is a git repository and building a
metadata URL from that URL is not that worth it). So when users add a
repository, they add the metadata url. What do you think?

 Cheers

On Mon, Aug 1, 2016 at 9:29 AM, Alessandro Pasotti 
wrote:

> On Sat, Jul 30, 2016 at 10:39 PM, Akbar Gumbira 
> wrote:
>
>> *What did you get done this week?*
>>
>>- *Implemented style resource handler*
>>
>> User can now also put style (qml file) in the collection.
>>
>>
>>- *Fixed issues on windows*
>>
>> It should be testable now on Windows. There is an issue on windows while
>> downloading the repository using ssh. I haven't prevented that (e.g
>> restricting to use https only when adding a repository in the settings -
>> but I guess it's easier to do this rather than checking if the host has ssh
>> installed)
>>
>>
>>- *Wrote some documentation* (
>>http://www.akbargumbira.com/qgis_resources_sharing/)
>>
>>
>>- *Implemented file system handler*
>>
>> User can now use repository in the local file system. The URI should have
>> this format: file://. On Windows it looks something
>> like this *file://C:/home/akbar/repo. *On linux:
>> *file:///home/akbar/repo*.
>> This could be useful for:
>>
>>
>>- Testing if the repository works fine before pushing to
>>   github/bitbucket (user can try to add a repo in the stting pointing to 
>> the
>>   filesystem)
>>   - In case someone wants to use private repo, they can pull it to
>>   their local machine, and just use that local clone
>>   - Sharing the repository on the network
>>
>> *What do you plan on doing next week?*
>>
>>- Documentation and unittests
>>- Using QGIS authentication system in the settings tab when adding a
>>repository (this would be easy in the case of fetching metadata from the
>>repository, but might be tricky for cloning/pulling repository itself)
>>- Ale and I had a talk on friday and zip repository handler might be
>>useful for users (i.e the author makes a zip of the repository on publish
>>it somewhere online). With the current implementation of repository 
>> handler
>>(the metadata must be inside the repository itself), this might take some
>>effort to implement. We don't want to download the whole zip just to get
>>the metadata file. Ale's idea is to separate the metadata and in the
>>metadata, additional things has to be defined: the protocol (e.g git, file
>>system handler, or ftp, scp maybe in the future), and the location of the
>>repository itself. I am a bit hesitant with this idea from the beginning
>>(Ale already suggested this before :)) as I think the metadata and the
>>repository must be in the same directory. It's nice to not have the
>>protocol and the location of the repository statically defined. As I
>>mentioned earlier with file system handler, user now can clone private
>>repository in github/bitbucket and use file system handler. There's no 
>> need
>>to change the metadata of the repository before using it (this is needed 
>> if
>>we want to take Ale's idea). Anyway, I'll think about this over the 
>> weekend
>>and see if I could im

Re: [Qgis-developer] Report 10 - QGIS Resources Sharing Tools

[Alessandro Pasotti]

On Sat, Jul 30, 2016 at 10:39 PM, Akbar Gumbira 
wrote:

> *What did you get done this week?*
>
>- *Implemented style resource handler*
>
> User can now also put style (qml file) in the collection.
>
>
>- *Fixed issues on windows*
>
> It should be testable now on Windows. There is an issue on windows while
> downloading the repository using ssh. I haven't prevented that (e.g
> restricting to use https only when adding a repository in the settings -
> but I guess it's easier to do this rather than checking if the host has ssh
> installed)
>
>
>- *Wrote some documentation* (
>http://www.akbargumbira.com/qgis_resources_sharing/)
>
>
>- *Implemented file system handler*
>
> User can now use repository in the local file system. The URI should have
> this format: file://. On Windows it looks something
> like this *file://C:/home/akbar/repo. *On linux: *file:///home/akbar/repo*
> .
> This could be useful for:
>
>
>- Testing if the repository works fine before pushing to
>   github/bitbucket (user can try to add a repo in the stting pointing to 
> the
>   filesystem)
>   - In case someone wants to use private repo, they can pull it to
>   their local machine, and just use that local clone
>   - Sharing the repository on the network
>
> *What do you plan on doing next week?*
>
>- Documentation and unittests
>- Using QGIS authentication system in the settings tab when adding a
>repository (this would be easy in the case of fetching metadata from the
>repository, but might be tricky for cloning/pulling repository itself)
>- Ale and I had a talk on friday and zip repository handler might be
>useful for users (i.e the author makes a zip of the repository on publish
>it somewhere online). With the current implementation of repository handler
>(the metadata must be inside the repository itself), this might take some
>effort to implement. We don't want to download the whole zip just to get
>the metadata file. Ale's idea is to separate the metadata and in the
>metadata, additional things has to be defined: the protocol (e.g git, file
>system handler, or ftp, scp maybe in the future), and the location of the
>repository itself. I am a bit hesitant with this idea from the beginning
>(Ale already suggested this before :)) as I think the metadata and the
>repository must be in the same directory. It's nice to not have the
>protocol and the location of the repository statically defined. As I
>mentioned earlier with file system handler, user now can clone private
>repository in github/bitbucket and use file system handler. There's no need
>to change the metadata of the repository before using it (this is needed if
>we want to take Ale's idea). Anyway, I'll think about this over the weekend
>and see if I could implement this on time and still have time to polish
>things up.
>
>
> *Are you blocked on anything?*
> No, just need to have more opinions on the situation I mentioned above.
>
> Have a nice weekend. Cheers!
> --
>


Thanks for your report Akbar!

I'm sorry if you've not taken into account my repeated calls to keep
authentication and proxy support in the primary goals. As I explained you
this is very important for some companies and organizations that needs
authenticated endpoints in their networks.

If this plugin will not use this feature, the a.m. organization will not be
able to use it (and they will not invest any time or resources in its
maintainance or future development), they will just build their own sharing
solution.

For this reason I've been pushing since the very beginning to use
QgsNetworkAccessManager for all network calls, in order to be able to use
QGIS proxy settings and authentication plugins.

For the zip/tar file handler (or other yet to be developed handlers), I
believe that unless we end up with a centralized index (like plugin manager
does with XML file), we really need to separate the metadata protocol from
the payload protocol.

Of course, this should be an optional override: if there is no change of
protocol for a particular collection, the handler will just use the repo's
protocol, but for the ZIP file if we do not want to download the whole zip
just to be able to read the metadata, there is no other way than to provide
a distinct metadata URL.

Additionally, I believe that providing a ZIP handler is very important,
since it will likely become the most popular way to share resources,
everybody is able to build a zip file and put it online, without even
knowing what GIT is.


The algorithm could be:

# Browsing:
- for each the repo url:
   - if it does not point to a metadata.ini file
  - get the metadata address from the URL
   - if we have an handler to retrieve the metadata.ini
  - retrieve the metadata.ini
   - else
  - error

#Installing
- if the metadata does not contain a download URL for the collection
- build the collection download URL from