Future of fedora-packages

[Clement Verna]

Hi all,

fedora-packages [0] code base is showing its age. The code base and
the technology stack  (Turbogears2 [1] web framework and the Moksha
[2] middleware) is currently not ready for Python3 and I am not
planning to do the work required to make it Python3 compatible, so the
application will stop working when Fedora 29 is EOL.

In order to keep the service running, I have started a Proof Of
Concept (fedora-search [3]) to replace the backend of the application.
Fedora-search would be a REST API service offering full test search
API. Such a service would then be available for other application to
use, fedora-packages would then become a frontend only application
using the service provided by fedora-search.

While the POC shows that this is a viable solution, I don't think that
we should be proceeding that way, for the simple reason that this add
yet another code base to maintain, I think we should use this
opportunity to consider using Elasticsearch instead of maintaining our
own "search engine".

I think that Elasticsearch offers quite a few advantages :
  - Powerful Query language
  - Python bindings
  - Javascript bindings
  - Can be deployed in our infrastructure or used as a service
  - Can be useful for other applications ( docs.fp.o, pagure, ??)

So what is the general feeling about using Elasticsearch in our
infrastructure ? Should we look at deploying a cluster in our infra /
Should we approach the Council to see if we can get founding to have
this service hosted by Elastic ?

Thanks
Clément

[0] - https://apps.fedoraproject.org/packages/
[1] - http://www.turbogears.org/
[2] - https://mokshaproject.github.io/mokshaproject.net/
[3] - https://github.com/fedora-infra/fedora-search
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Ryan Lerch]

On Wed, 27 Feb 2019 at 05:39, Clement Verna 
wrote:

> Hi all,
>
> fedora-packages [0] code base is showing its age. The code base and
> the technology stack  (Turbogears2 [1] web framework and the Moksha
> [2] middleware) is currently not ready for Python3 and I am not
> planning to do the work required to make it Python3 compatible, so the
> application will stop working when Fedora 29 is EOL.
>
> In order to keep the service running, I have started a Proof Of
> Concept (fedora-search [3]) to replace the backend of the application.
> Fedora-search would be a REST API service offering full test search
> API. Such a service would then be available for other application to
> use, fedora-packages would then become a frontend only application
> using the service provided by fedora-search.
>
> While the POC shows that this is a viable solution, I don't think that
> we should be proceeding that way, for the simple reason that this add
> yet another code base to maintain, I think we should use this
> opportunity to consider using Elasticsearch instead of maintaining our
> own "search engine".
>
> I think that Elasticsearch offers quite a few advantages :
>   - Powerful Query language
>   - Python bindings
>   - Javascript bindings
>   - Can be deployed in our infrastructure or used as a service
>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>
> So what is the general feeling about using Elasticsearch in our
> infrastructure ? Should we look at deploying a cluster in our infra /
> Should we approach the Council to see if we can get founding to have
> this service hosted by Elastic ?
>
> Thanks
> Clément


>From an information point of view, the package-centric view of
Fedora-packages is quite similar to the pagure dist-git. Would it worth
investgating merging the front-end functionality of packages (lists of
builds, bugs, updates, etc) into pagure dist-git, and retiring the Fedora
packages front end?

—ry

>
>
> [0] - https://apps.fedoraproject.org/packages/
> [1] - http://www.turbogears.org/
> [2] - https://mokshaproject.github.io/mokshaproject.net/
> [3] - https://github.com/fedora-infra/fedora-search
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to
> infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
>
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

On Tue, 26 Feb 2019 at 23:52, Ryan Lerch  wrote:
>
>
>
> On Wed, 27 Feb 2019 at 05:39, Clement Verna  wrote:
>>
>> Hi all,
>>
>> fedora-packages [0] code base is showing its age. The code base and
>> the technology stack  (Turbogears2 [1] web framework and the Moksha
>> [2] middleware) is currently not ready for Python3 and I am not
>> planning to do the work required to make it Python3 compatible, so the
>> application will stop working when Fedora 29 is EOL.
>>
>> In order to keep the service running, I have started a Proof Of
>> Concept (fedora-search [3]) to replace the backend of the application.
>> Fedora-search would be a REST API service offering full test search
>> API. Such a service would then be available for other application to
>> use, fedora-packages would then become a frontend only application
>> using the service provided by fedora-search.
>>
>> While the POC shows that this is a viable solution, I don't think that
>> we should be proceeding that way, for the simple reason that this add
>> yet another code base to maintain, I think we should use this
>> opportunity to consider using Elasticsearch instead of maintaining our
>> own "search engine".
>>
>> I think that Elasticsearch offers quite a few advantages :
>>   - Powerful Query language
>>   - Python bindings
>>   - Javascript bindings
>>   - Can be deployed in our infrastructure or used as a service
>>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>>
>> So what is the general feeling about using Elasticsearch in our
>> infrastructure ? Should we look at deploying a cluster in our infra /
>> Should we approach the Council to see if we can get founding to have
>> this service hosted by Elastic ?
>>
>> Thanks
>> Clément
>
>
> From an information point of view, the package-centric view of 
> Fedora-packages is quite similar to the pagure dist-git. Would it worth 
> investgating merging the front-end functionality of packages (lists of 
> builds, bugs, updates, etc) into pagure dist-git, and retiring the Fedora 
> packages front end?

+1 I think that would be a good way to explore

>
> —ry
>>
>>
>>
>> [0] - https://apps.fedoraproject.org/packages/
>> [1] - http://www.turbogears.org/
>> [2] - https://mokshaproject.github.io/mokshaproject.net/
>> [3] - https://github.com/fedora-infra/fedora-search
>> ___
>> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
>> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: 
>> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Vít Ondruch]

Dne 26. 02. 19 v 23:42 Ryan Lerch napsal(a):
>
>
> On Wed, 27 Feb 2019 at 05:39, Clement Verna  > wrote:
>
> Hi all,
>
> fedora-packages [0] code base is showing its age. The code base and
> the technology stack  (Turbogears2 [1] web framework and the Moksha
> [2] middleware) is currently not ready for Python3 and I am not
> planning to do the work required to make it Python3 compatible, so the
> application will stop working when Fedora 29 is EOL.
>
> In order to keep the service running, I have started a Proof Of
> Concept (fedora-search [3]) to replace the backend of the application.
>

Backend of fedora-packages is mdapi, isn't it? What would be the difference?


> Fedora-search would be a REST API service offering full test search
> API. Such a service would then be available for other application to
> use, fedora-packages would then become a frontend only application
> using the service provided by fedora-search.
>
> While the POC shows that this is a viable solution, I don't think that
> we should be proceeding that way, for the simple reason that this add
> yet another code base to maintain, I think we should use this
> opportunity to consider using Elasticsearch instead of maintaining our
> own "search engine".
>
> I think that Elasticsearch offers quite a few advantages :
>   - Powerful Query language
>   - Python bindings
>   - Javascript bindings
>   - Can be deployed in our infrastructure or used as a service
>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>
> So what is the general feeling about using Elasticsearch in our
> infrastructure ? Should we look at deploying a cluster in our infra /
> Should we approach the Council to see if we can get founding to have
> this service hosted by Elastic ?
>
> Thanks
> Clément
>
>
> From an information point of view, the package-centric view of
> Fedora-packages is quite similar to the pagure dist-git. Would it
> worth investgating merging the front-end functionality of packages
> (lists of builds, bugs, updates, etc) into pagure dist-git, and
> retiring the Fedora packages front end?


+1


V.


>
> —ry
>
>
>
> [0] - https://apps.fedoraproject.org/packages/
> [1] - http://www.turbogears.org/
> [2] - https://mokshaproject.github.io/mokshaproject.net/
> [3] - https://github.com/fedora-infra/fedora-search
> ___
> infrastructure mailing list --
> infrastructure@lists.fedoraproject.org
> 
> To unsubscribe send an email to
> infrastructure-le...@lists.fedoraproject.org
> 
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
>
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Neal Gompa]

On Tue, Feb 26, 2019 at 5:43 PM Ryan Lerch  wrote:
>
> On Wed, 27 Feb 2019 at 05:39, Clement Verna  wrote:
>>
>> Hi all,
>>
>> fedora-packages [0] code base is showing its age. The code base and
>> the technology stack  (Turbogears2 [1] web framework and the Moksha
>> [2] middleware) is currently not ready for Python3 and I am not
>> planning to do the work required to make it Python3 compatible, so the
>> application will stop working when Fedora 29 is EOL.
>>
>> In order to keep the service running, I have started a Proof Of
>> Concept (fedora-search [3]) to replace the backend of the application.
>> Fedora-search would be a REST API service offering full test search
>> API. Such a service would then be available for other application to
>> use, fedora-packages would then become a frontend only application
>> using the service provided by fedora-search.
>>
>> While the POC shows that this is a viable solution, I don't think that
>> we should be proceeding that way, for the simple reason that this add
>> yet another code base to maintain, I think we should use this
>> opportunity to consider using Elasticsearch instead of maintaining our
>> own "search engine".
>>
>> I think that Elasticsearch offers quite a few advantages :
>>   - Powerful Query language
>>   - Python bindings
>>   - Javascript bindings
>>   - Can be deployed in our infrastructure or used as a service
>>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>>
>> So what is the general feeling about using Elasticsearch in our
>> infrastructure ? Should we look at deploying a cluster in our infra /
>> Should we approach the Council to see if we can get founding to have
>> this service hosted by Elastic ?
>>
>> Thanks
>> Clément
>
>
> From an information point of view, the package-centric view of 
> Fedora-packages is quite similar to the pagure dist-git. Would it worth 
> investgating merging the front-end functionality of packages (lists of 
> builds, bugs, updates, etc) into pagure dist-git, and retiring the Fedora 
> packages front end?
>

How would you propose we'd allow people to search by binary packages
and seeing contents, changelog, and seeing which versions are shipped
in releases?

I don't think it would be a good idea to merge the two views, because
it really doesn't make this any easier. The package search is a
user-focused view into the package collection, and is a helpful
reference (when it works!). I think it still serves a purpose to have
a dedicated frontend.


-- 
真実はいつも一つ！/ Always, there's only one truth!
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Pierre-Yves Chibon]

On Tue, Feb 26, 2019 at 08:38:30PM +0100, Clement Verna wrote:
> Hi all,
> 
> fedora-packages [0] code base is showing its age. The code base and
> the technology stack  (Turbogears2 [1] web framework and the Moksha
> [2] middleware) is currently not ready for Python3 and I am not
> planning to do the work required to make it Python3 compatible, so the
> application will stop working when Fedora 29 is EOL.
> 
> In order to keep the service running, I have started a Proof Of
> Concept (fedora-search [3]) to replace the backend of the application.
> Fedora-search would be a REST API service offering full test search
> API. Such a service would then be available for other application to
> use, fedora-packages would then become a frontend only application
> using the service provided by fedora-search.
> 
> While the POC shows that this is a viable solution, I don't think that
> we should be proceeding that way, for the simple reason that this add
> yet another code base to maintain, I think we should use this
> opportunity to consider using Elasticsearch instead of maintaining our
> own "search engine".
> 
> I think that Elasticsearch offers quite a few advantages :
>   - Powerful Query language
>   - Python bindings
>   - Javascript bindings
>   - Can be deployed in our infrastructure or used as a service
>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> 
> So what is the general feeling about using Elasticsearch in our
> infrastructure ? Should we look at deploying a cluster in our infra /
> Should we approach the Council to see if we can get founding to have
> this service hosted by Elastic ?

We look at Elasticsearch at one point (I believe Aurélien was even looking at
setting up an instance in our cloud) but it came down to two aspects:
- setting it up is another application to maintain
- an application we have no expertise on

On the other side, fedora-search is:
- setting up another application to maintain
- an application we have expertise on

Asking Council to get founding for a hosted instance would resolve both of
points for Elasticsearch without the time implication of developing it
ourself.
In both case, there will be some time implication of moving to the new system,
so that's not a tie-breaker.


Pierre
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

On Wed, 27 Feb 2019 at 10:27, Vít Ondruch  wrote:
>
>
> Dne 26. 02. 19 v 23:42 Ryan Lerch napsal(a):
>
>
>
> On Wed, 27 Feb 2019 at 05:39, Clement Verna  wrote:
>>
>> Hi all,
>>
>> fedora-packages [0] code base is showing its age. The code base and
>> the technology stack  (Turbogears2 [1] web framework and the Moksha
>> [2] middleware) is currently not ready for Python3 and I am not
>> planning to do the work required to make it Python3 compatible, so the
>> application will stop working when Fedora 29 is EOL.
>>
>> In order to keep the service running, I have started a Proof Of
>> Concept (fedora-search [3]) to replace the backend of the application.
>
>
> Backend of fedora-packages is mdapi, isn't it? What would be the difference?

mdapi is one of the service used to populate the xapian search index.
The fedora-packages backend is providing the API to the xapian
database to perform the full text search. If we were to use
elasticsearch we would still need to have an indexing script but we
could rely on elasticsearch API and query language for the search
capabilities.

>
>
>> Fedora-search would be a REST API service offering full test search
>> API. Such a service would then be available for other application to
>> use, fedora-packages would then become a frontend only application
>> using the service provided by fedora-search.
>>
>> While the POC shows that this is a viable solution, I don't think that
>> we should be proceeding that way, for the simple reason that this add
>> yet another code base to maintain, I think we should use this
>> opportunity to consider using Elasticsearch instead of maintaining our
>> own "search engine".
>>
>> I think that Elasticsearch offers quite a few advantages :
>>   - Powerful Query language
>>   - Python bindings
>>   - Javascript bindings
>>   - Can be deployed in our infrastructure or used as a service
>>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>>
>> So what is the general feeling about using Elasticsearch in our
>> infrastructure ? Should we look at deploying a cluster in our infra /
>> Should we approach the Council to see if we can get founding to have
>> this service hosted by Elastic ?
>>
>> Thanks
>> Clément
>
>
> From an information point of view, the package-centric view of 
> Fedora-packages is quite similar to the pagure dist-git. Would it worth 
> investgating merging the front-end functionality of packages (lists of 
> builds, bugs, updates, etc) into pagure dist-git, and retiring the Fedora 
> packages front end?
>
>
> +1
>
>
> V.
>
>
>
> —ry
>>
>>
>>
>> [0] - https://apps.fedoraproject.org/packages/
>> [1] - http://www.turbogears.org/
>> [2] - https://mokshaproject.github.io/mokshaproject.net/
>> [3] - https://github.com/fedora-infra/fedora-search
>> ___
>> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
>> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: 
>> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
>
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Stephen John Smoogen]

On Tue, 26 Feb 2019 at 14:39, Clement Verna  wrote:
>
> Hi all,
>
> fedora-packages [0] code base is showing its age. The code base and
> the technology stack  (Turbogears2 [1] web framework and the Moksha
> [2] middleware) is currently not ready for Python3 and I am not
> planning to do the work required to make it Python3 compatible, so the
> application will stop working when Fedora 29 is EOL.
>
> In order to keep the service running, I have started a Proof Of
> Concept (fedora-search [3]) to replace the backend of the application.
> Fedora-search would be a REST API service offering full test search
> API. Such a service would then be available for other application to
> use, fedora-packages would then become a frontend only application
> using the service provided by fedora-search.
>
> While the POC shows that this is a viable solution, I don't think that
> we should be proceeding that way, for the simple reason that this add
> yet another code base to maintain, I think we should use this
> opportunity to consider using Elasticsearch instead of maintaining our
> own "search engine".
>

The main issues to getting elasticsearch working in the past was the following:

1 The number of systems needed to make it work. There is a large
difference from their 'proof-of-concept see how great this is' to 'ok
you want to do anything with load' setups in everything from storage
to number of search nodes to network speeds. [The number of hardware
for the data we have was to start with 5-8 dedicated Dell systems,
some amount of shared fast storage, and N virtual machines with a
10-40GB backbone.. or throwing all of Fedora Infrastructure at once
into the cloud.. because the feed it from PHX2 to the cloud is
expensive.]

2. Packaging of elasticsearch was a mess. At the time we had rules
that all packages needed to be packaged in Fedora and follow Fedora
packaging rules. [This one has been relaxed.]

3. Running of elasticsearch was a large service in itself. It doesn't
take care of itself and we would need one or more people who know it
well to keep it running. [This goes down the ladder.. the logstash
backends are also full services.. ] Most of that was written in Java
which no one on the team at the time had good experiences with.

4. A kibana/elasticsearch query expert. Just like any database, most
of the queries you can make are the worse kind. They will take a lot
more CPU/memory/time than they should making just grepping for data
faster.

However that is 3-5 years ago.. so a lot has changed since then.


> I think that Elasticsearch offers quite a few advantages :
>   - Powerful Query language
>   - Python bindings
>   - Javascript bindings
>   - Can be deployed in our infrastructure or used as a service
>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>
> So what is the general feeling about using Elasticsearch in our
> infrastructure ? Should we look at deploying a cluster in our infra /
> Should we approach the Council to see if we can get founding to have
> this service hosted by Elastic ?
>
> Thanks
> Clément
>
> [0] - https://apps.fedoraproject.org/packages/
> [1] - http://www.turbogears.org/
> [2] - https://mokshaproject.github.io/mokshaproject.net/
> [3] - https://github.com/fedora-infra/fedora-search
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org



-- 
Stephen J Smoogen.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

On Wed, 27 Feb 2019 at 13:27, Stephen John Smoogen  wrote:
>
> On Tue, 26 Feb 2019 at 14:39, Clement Verna  wrote:
> >
> > Hi all,
> >
> > fedora-packages [0] code base is showing its age. The code base and
> > the technology stack  (Turbogears2 [1] web framework and the Moksha
> > [2] middleware) is currently not ready for Python3 and I am not
> > planning to do the work required to make it Python3 compatible, so the
> > application will stop working when Fedora 29 is EOL.
> >
> > In order to keep the service running, I have started a Proof Of
> > Concept (fedora-search [3]) to replace the backend of the application.
> > Fedora-search would be a REST API service offering full test search
> > API. Such a service would then be available for other application to
> > use, fedora-packages would then become a frontend only application
> > using the service provided by fedora-search.
> >
> > While the POC shows that this is a viable solution, I don't think that
> > we should be proceeding that way, for the simple reason that this add
> > yet another code base to maintain, I think we should use this
> > opportunity to consider using Elasticsearch instead of maintaining our
> > own "search engine".
> >
>
> The main issues to getting elasticsearch working in the past was the 
> following:
>
> 1 The number of systems needed to make it work. There is a large
> difference from their 'proof-of-concept see how great this is' to 'ok
> you want to do anything with load' setups in everything from storage
> to number of search nodes to network speeds. [The number of hardware
> for the data we have was to start with 5-8 dedicated Dell systems,
> some amount of shared fast storage, and N virtual machines with a
> 10-40GB backbone.. or throwing all of Fedora Infrastructure at once
> into the cloud.. because the feed it from PHX2 to the cloud is
> expensive.]
>
> 2. Packaging of elasticsearch was a mess. At the time we had rules
> that all packages needed to be packaged in Fedora and follow Fedora
> packaging rules. [This one has been relaxed.]
>
> 3. Running of elasticsearch was a large service in itself. It doesn't
> take care of itself and we would need one or more people who know it
> well to keep it running. [This goes down the ladder.. the logstash
> backends are also full services.. ] Most of that was written in Java
> which no one on the team at the time had good experiences with.
>
> 4. A kibana/elasticsearch query expert. Just like any database, most
> of the queries you can make are the worse kind. They will take a lot
> more CPU/memory/time than they should making just grepping for data
> faster.
>
> However that is 3-5 years ago.. so a lot has changed since then.

Thanks smooge for feeling my knowledge gap on what was looked at
previously, I feel that a lot of the issues would actually be solved
if we were to outsource the service and have someone else maintain the
cluster for us. If that's the case I would be happy to open a council
ticket to start the conversation about this possibility .

>
>
> > I think that Elasticsearch offers quite a few advantages :
> >   - Powerful Query language
> >   - Python bindings
> >   - Javascript bindings
> >   - Can be deployed in our infrastructure or used as a service
> >   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> >
> > So what is the general feeling about using Elasticsearch in our
> > infrastructure ? Should we look at deploying a cluster in our infra /
> > Should we approach the Council to see if we can get founding to have
> > this service hosted by Elastic ?
> >
> > Thanks
> > Clément
> >
> > [0] - https://apps.fedoraproject.org/packages/
> > [1] - http://www.turbogears.org/
> > [2] - https://mokshaproject.github.io/mokshaproject.net/
> > [3] - https://github.com/fedora-infra/fedora-search
> > ___
> > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
>
>
>
> --
> Stephen J Smoogen.
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.

Re: Future of fedora-packages

[Mikolaj Izdebski]

On Wed, Feb 27, 2019 at 1:20 PM Stephen John Smoogen  wrote:
> 2. Packaging of elasticsearch was a mess. At the time we had rules
> that all packages needed to be packaged in Fedora and follow Fedora
> packaging rules. [This one has been relaxed.]

I just want to point out that Elasticsearch has been packaged [1] in
Fedora for more than 4 years.

https://src.fedoraproject.org/rpms/elasticsearch

--
Mikolaj Izdebski
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Antonette Caldwell]

I looked at the version of Elasticsearch via sources and spec file, and I know 
that Elasticsearch is on 6.0 right now. It is true that we do need an expert in 
elasticsearch in order to move forward on using elasticsearch as an api if 
agreed on using elasticsearch.

I am currently working on Elasticsearch on another project and learning more 
about Elasticsearch, but this is my first time actually using it. Are there any 
plans to upgrade Elasticsearch to an updated version?
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Jim Perrin]

How much heresy is involved in us using Amazon's elasticsearch service
for this, so that we don't have yet-another-thing to maintain?

On 2/27/19 4:19 AM, Stephen John Smoogen wrote:
> On Tue, 26 Feb 2019 at 14:39, Clement Verna  wrote:
>>
>> Hi all,
>>
>> fedora-packages [0] code base is showing its age. The code base and
>> the technology stack  (Turbogears2 [1] web framework and the Moksha
>> [2] middleware) is currently not ready for Python3 and I am not
>> planning to do the work required to make it Python3 compatible, so the
>> application will stop working when Fedora 29 is EOL.
>>
>> In order to keep the service running, I have started a Proof Of
>> Concept (fedora-search [3]) to replace the backend of the application.
>> Fedora-search would be a REST API service offering full test search
>> API. Such a service would then be available for other application to
>> use, fedora-packages would then become a frontend only application
>> using the service provided by fedora-search.
>>
>> While the POC shows that this is a viable solution, I don't think that
>> we should be proceeding that way, for the simple reason that this add
>> yet another code base to maintain, I think we should use this
>> opportunity to consider using Elasticsearch instead of maintaining our
>> own "search engine".
>>
> 
> The main issues to getting elasticsearch working in the past was the 
> following:
> 
> 1 The number of systems needed to make it work. There is a large
> difference from their 'proof-of-concept see how great this is' to 'ok
> you want to do anything with load' setups in everything from storage
> to number of search nodes to network speeds. [The number of hardware
> for the data we have was to start with 5-8 dedicated Dell systems,
> some amount of shared fast storage, and N virtual machines with a
> 10-40GB backbone.. or throwing all of Fedora Infrastructure at once
> into the cloud.. because the feed it from PHX2 to the cloud is
> expensive.]
> 
> 2. Packaging of elasticsearch was a mess. At the time we had rules
> that all packages needed to be packaged in Fedora and follow Fedora
> packaging rules. [This one has been relaxed.]
> 
> 3. Running of elasticsearch was a large service in itself. It doesn't
> take care of itself and we would need one or more people who know it
> well to keep it running. [This goes down the ladder.. the logstash
> backends are also full services.. ] Most of that was written in Java
> which no one on the team at the time had good experiences with.
> 
> 4. A kibana/elasticsearch query expert. Just like any database, most
> of the queries you can make are the worse kind. They will take a lot
> more CPU/memory/time than they should making just grepping for data
> faster.
> 
> However that is 3-5 years ago.. so a lot has changed since then.
> 
> 
>> I think that Elasticsearch offers quite a few advantages :
>>   - Powerful Query language
>>   - Python bindings
>>   - Javascript bindings
>>   - Can be deployed in our infrastructure or used as a service
>>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>>
>> So what is the general feeling about using Elasticsearch in our
>> infrastructure ? Should we look at deploying a cluster in our infra /
>> Should we approach the Council to see if we can get founding to have
>> this service hosted by Elastic ?
>>
>> Thanks
>> Clément
>>
>> [0] - https://apps.fedoraproject.org/packages/
>> [1] - http://www.turbogears.org/
>> [2] - https://mokshaproject.github.io/mokshaproject.net/
>> [3] - https://github.com/fedora-infra/fedora-search
>> ___
>> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
>> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: 
>> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> 
> 
> 
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Stephen John Smoogen]

On Wed, 27 Feb 2019 at 14:36, Mikolaj Izdebski  wrote:
>
> On Wed, Feb 27, 2019 at 1:20 PM Stephen John Smoogen  wrote:
> > 2. Packaging of elasticsearch was a mess. At the time we had rules
> > that all packages needed to be packaged in Fedora and follow Fedora
> > packaging rules. [This one has been relaxed.]
>
> I just want to point out that Elasticsearch has been packaged [1] in
> Fedora for more than 4 years.
>
> https://src.fedoraproject.org/rpms/elasticsearch

The version I am seeing there is 1.7.5 and the version on github is
6.6.1 .. i didn't know if that was a 'dont even think about using
that'




-- 
Stephen J Smoogen.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Neal Gompa]

On Wed, Feb 27, 2019 at 5:56 PM Stephen John Smoogen  wrote:
>
> On Wed, 27 Feb 2019 at 14:36, Mikolaj Izdebski  wrote:
> >
> > On Wed, Feb 27, 2019 at 1:20 PM Stephen John Smoogen  
> > wrote:
> > > 2. Packaging of elasticsearch was a mess. At the time we had rules
> > > that all packages needed to be packaged in Fedora and follow Fedora
> > > packaging rules. [This one has been relaxed.]
> >
> > I just want to point out that Elasticsearch has been packaged [1] in
> > Fedora for more than 4 years.
> >
> > https://src.fedoraproject.org/rpms/elasticsearch
>
> The version I am seeing there is 1.7.5 and the version on github is
> 6.6.1 .. i didn't know if that was a 'dont even think about using
> that'
>

Upstream skipped a bunch of versions. They went from 1.x to 2.x to 5.x
and now 6.x with 7.x in development.

I forgot the exact reason for this, but it had something to do with
aligning the versions across all of Elastic's software.



-- 
真実はいつも一つ！/ Always, there's only one truth!
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Stephen John Smoogen]

On Wed, 27 Feb 2019 at 16:05, Jim Perrin  wrote:
>
> How much heresy is involved in us using Amazon's elasticsearch service
> for this, so that we don't have yet-another-thing to maintain?
>

I was wondering how much data are we looking to shove there, does that
data need to be 'protected', and how fast do we need it to be for us
to talk back and forth to the cloud. The heresy side I don't have any
say in..


> On 2/27/19 4:19 AM, Stephen John Smoogen wrote:
> > On Tue, 26 Feb 2019 at 14:39, Clement Verna  
> > wrote:
> >>
> >> Hi all,
> >>
> >> fedora-packages [0] code base is showing its age. The code base and
> >> the technology stack  (Turbogears2 [1] web framework and the Moksha
> >> [2] middleware) is currently not ready for Python3 and I am not
> >> planning to do the work required to make it Python3 compatible, so the
> >> application will stop working when Fedora 29 is EOL.
> >>
> >> In order to keep the service running, I have started a Proof Of
> >> Concept (fedora-search [3]) to replace the backend of the application.
> >> Fedora-search would be a REST API service offering full test search
> >> API. Such a service would then be available for other application to
> >> use, fedora-packages would then become a frontend only application
> >> using the service provided by fedora-search.
> >>
> >> While the POC shows that this is a viable solution, I don't think that
> >> we should be proceeding that way, for the simple reason that this add
> >> yet another code base to maintain, I think we should use this
> >> opportunity to consider using Elasticsearch instead of maintaining our
> >> own "search engine".
> >>
> >
> > The main issues to getting elasticsearch working in the past was the 
> > following:
> >
> > 1 The number of systems needed to make it work. There is a large
> > difference from their 'proof-of-concept see how great this is' to 'ok
> > you want to do anything with load' setups in everything from storage
> > to number of search nodes to network speeds. [The number of hardware
> > for the data we have was to start with 5-8 dedicated Dell systems,
> > some amount of shared fast storage, and N virtual machines with a
> > 10-40GB backbone.. or throwing all of Fedora Infrastructure at once
> > into the cloud.. because the feed it from PHX2 to the cloud is
> > expensive.]
> >
> > 2. Packaging of elasticsearch was a mess. At the time we had rules
> > that all packages needed to be packaged in Fedora and follow Fedora
> > packaging rules. [This one has been relaxed.]
> >
> > 3. Running of elasticsearch was a large service in itself. It doesn't
> > take care of itself and we would need one or more people who know it
> > well to keep it running. [This goes down the ladder.. the logstash
> > backends are also full services.. ] Most of that was written in Java
> > which no one on the team at the time had good experiences with.
> >
> > 4. A kibana/elasticsearch query expert. Just like any database, most
> > of the queries you can make are the worse kind. They will take a lot
> > more CPU/memory/time than they should making just grepping for data
> > faster.
> >
> > However that is 3-5 years ago.. so a lot has changed since then.
> >
> >
> >> I think that Elasticsearch offers quite a few advantages :
> >>   - Powerful Query language
> >>   - Python bindings
> >>   - Javascript bindings
> >>   - Can be deployed in our infrastructure or used as a service
> >>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> >>
> >> So what is the general feeling about using Elasticsearch in our
> >> infrastructure ? Should we look at deploying a cluster in our infra /
> >> Should we approach the Council to see if we can get founding to have
> >> this service hosted by Elastic ?
> >>
> >> Thanks
> >> Clément
> >>
> >> [0] - https://apps.fedoraproject.org/packages/
> >> [1] - http://www.turbogears.org/
> >> [2] - https://mokshaproject.github.io/mokshaproject.net/
> >> [3] - https://github.com/fedora-infra/fedora-search
> >> ___
> >> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> >> To unsubscribe send an email to 
> >> infrastructure-le...@lists.fedoraproject.org
> >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives: 
> >> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> >
> >
> >
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org



-- 
Stephen J Smoogen.
___

Re: Future of fedora-packages

[Todd Zullinger]

Neal Gompa wrote:
> On Wed, Feb 27, 2019 at 5:56 PM Stephen John Smoogen  wrote:
>>
>> On Wed, 27 Feb 2019 at 14:36, Mikolaj Izdebski  wrote:
>>>
>>> On Wed, Feb 27, 2019 at 1:20 PM Stephen John Smoogen  
>>> wrote:
 2. Packaging of elasticsearch was a mess. At the time we had rules
 that all packages needed to be packaged in Fedora and follow Fedora
 packaging rules. [This one has been relaxed.]
>>>
>>> I just want to point out that Elasticsearch has been packaged [1] in
>>> Fedora for more than 4 years.
>>>
>>> https://src.fedoraproject.org/rpms/elasticsearch
>>
>> The version I am seeing there is 1.7.5 and the version on github is
>> 6.6.1 .. i didn't know if that was a 'dont even think about using
>> that'
> 
> Upstream skipped a bunch of versions. They went from 1.x to 2.x to 5.x
> and now 6.x with 7.x in development.
> 
> I forgot the exact reason for this, but it had something to do with
> aligning the versions across all of Elastic's software.

Though it's still worth noting that 1.7.5 has been EOL for 2
years (2017-01-16, per https://www.elastic.co/support/eol).
There's around 15 CVE's since that release (whether 1.7.x
is vulnerable to any/all of them is another matter).

It doesn't seem like the Fedora packages are being actively
maintained.  There's been no substanative commits in the
past 3 years.

-- 
Todd


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Antonette Caldwell]

I would like to take on maintaining the package, however, I do not have 
adequate experience in creating and maintaining as such. I have looked at the 
rpm packaging guidelines, and I can see what to do, but actually doing it is a 
little bit different. I understand everyone has different methods in completing 
the same outcome, which is creating and maintaining packages.

I am learning, however, within a mock environment, but I do not sufficient 
permissions to push the package anyway. If it is possible to go forward with 
Elasticsearch, I would be interested in maintaining but I would need a 
co-maintainer for assistance.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

On Wed, 27 Feb 2019 at 22:11, Jim Perrin  wrote:
>
> How much heresy is involved in us using Amazon's elasticsearch service
> for this, so that we don't have yet-another-thing to maintain?

If we want to proceed with elasticsearch I think that this is indeed a
good way forward. What would it take to use our current AWS account to
setup a test cluster ?

>
> On 2/27/19 4:19 AM, Stephen John Smoogen wrote:
> > On Tue, 26 Feb 2019 at 14:39, Clement Verna  
> > wrote:
> >>
> >> Hi all,
> >>
> >> fedora-packages [0] code base is showing its age. The code base and
> >> the technology stack  (Turbogears2 [1] web framework and the Moksha
> >> [2] middleware) is currently not ready for Python3 and I am not
> >> planning to do the work required to make it Python3 compatible, so the
> >> application will stop working when Fedora 29 is EOL.
> >>
> >> In order to keep the service running, I have started a Proof Of
> >> Concept (fedora-search [3]) to replace the backend of the application.
> >> Fedora-search would be a REST API service offering full test search
> >> API. Such a service would then be available for other application to
> >> use, fedora-packages would then become a frontend only application
> >> using the service provided by fedora-search.
> >>
> >> While the POC shows that this is a viable solution, I don't think that
> >> we should be proceeding that way, for the simple reason that this add
> >> yet another code base to maintain, I think we should use this
> >> opportunity to consider using Elasticsearch instead of maintaining our
> >> own "search engine".
> >>
> >
> > The main issues to getting elasticsearch working in the past was the 
> > following:
> >
> > 1 The number of systems needed to make it work. There is a large
> > difference from their 'proof-of-concept see how great this is' to 'ok
> > you want to do anything with load' setups in everything from storage
> > to number of search nodes to network speeds. [The number of hardware
> > for the data we have was to start with 5-8 dedicated Dell systems,
> > some amount of shared fast storage, and N virtual machines with a
> > 10-40GB backbone.. or throwing all of Fedora Infrastructure at once
> > into the cloud.. because the feed it from PHX2 to the cloud is
> > expensive.]
> >
> > 2. Packaging of elasticsearch was a mess. At the time we had rules
> > that all packages needed to be packaged in Fedora and follow Fedora
> > packaging rules. [This one has been relaxed.]
> >
> > 3. Running of elasticsearch was a large service in itself. It doesn't
> > take care of itself and we would need one or more people who know it
> > well to keep it running. [This goes down the ladder.. the logstash
> > backends are also full services.. ] Most of that was written in Java
> > which no one on the team at the time had good experiences with.
> >
> > 4. A kibana/elasticsearch query expert. Just like any database, most
> > of the queries you can make are the worse kind. They will take a lot
> > more CPU/memory/time than they should making just grepping for data
> > faster.
> >
> > However that is 3-5 years ago.. so a lot has changed since then.
> >
> >
> >> I think that Elasticsearch offers quite a few advantages :
> >>   - Powerful Query language
> >>   - Python bindings
> >>   - Javascript bindings
> >>   - Can be deployed in our infrastructure or used as a service
> >>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> >>
> >> So what is the general feeling about using Elasticsearch in our
> >> infrastructure ? Should we look at deploying a cluster in our infra /
> >> Should we approach the Council to see if we can get founding to have
> >> this service hosted by Elastic ?
> >>
> >> Thanks
> >> Clément
> >>
> >> [0] - https://apps.fedoraproject.org/packages/
> >> [1] - http://www.turbogears.org/
> >> [2] - https://mokshaproject.github.io/mokshaproject.net/
> >> [3] - https://github.com/fedora-infra/fedora-search
> >> ___
> >> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> >> To unsubscribe send an email to 
> >> infrastructure-le...@lists.fedoraproject.org
> >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives: 
> >> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> >
> >
> >
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To

Re: Future of fedora-packages

[Clement Verna]

On Thu, 28 Feb 2019 at 00:06, Stephen John Smoogen  wrote:
>
> On Wed, 27 Feb 2019 at 16:05, Jim Perrin  wrote:
> >
> > How much heresy is involved in us using Amazon's elasticsearch service
> > for this, so that we don't have yet-another-thing to maintain?
> >
>
> I was wondering how much data are we looking to shove there, does that
> data need to be 'protected', and how fast do we need it to be for us
> to talk back and forth to the cloud. The heresy side I don't have any
> say in..

For fedora-packages we want to store documents that contains packages
informations (see the current structure used
https://github.com/fedora-infra/fedora-packages/blob/master/fedoracommunity/search/index.py#L241).
Currently in production we have 23849 documents in the xapian database
so I honestly don't think that will be much trouble for elasticsearch.
Writing to the cluster should be restricted and I think the search
service should be public, elasticsearch provides Security Privileges
(https://www.elastic.co/guide/en/x-pack/current/security-privileges.html)
that seems to fit with that idea.

Indexing does not have to be crazy fast, for example currently
fedora-packages indexing takes between 2 to 3 hours so I don't think
network latency will matter much here. Searching is a bit more
sensitive since users usually don't want to wait more than a seconds
or so to get a search results but if we use the elasticsearch
javascript library
(https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/index.html)
and handle the search in the frontend then it does not have to go via
our infrastructure.

>
>
> > On 2/27/19 4:19 AM, Stephen John Smoogen wrote:
> > > On Tue, 26 Feb 2019 at 14:39, Clement Verna  
> > > wrote:
> > >>
> > >> Hi all,
> > >>
> > >> fedora-packages [0] code base is showing its age. The code base and
> > >> the technology stack  (Turbogears2 [1] web framework and the Moksha
> > >> [2] middleware) is currently not ready for Python3 and I am not
> > >> planning to do the work required to make it Python3 compatible, so the
> > >> application will stop working when Fedora 29 is EOL.
> > >>
> > >> In order to keep the service running, I have started a Proof Of
> > >> Concept (fedora-search [3]) to replace the backend of the application.
> > >> Fedora-search would be a REST API service offering full test search
> > >> API. Such a service would then be available for other application to
> > >> use, fedora-packages would then become a frontend only application
> > >> using the service provided by fedora-search.
> > >>
> > >> While the POC shows that this is a viable solution, I don't think that
> > >> we should be proceeding that way, for the simple reason that this add
> > >> yet another code base to maintain, I think we should use this
> > >> opportunity to consider using Elasticsearch instead of maintaining our
> > >> own "search engine".
> > >>
> > >
> > > The main issues to getting elasticsearch working in the past was the 
> > > following:
> > >
> > > 1 The number of systems needed to make it work. There is a large
> > > difference from their 'proof-of-concept see how great this is' to 'ok
> > > you want to do anything with load' setups in everything from storage
> > > to number of search nodes to network speeds. [The number of hardware
> > > for the data we have was to start with 5-8 dedicated Dell systems,
> > > some amount of shared fast storage, and N virtual machines with a
> > > 10-40GB backbone.. or throwing all of Fedora Infrastructure at once
> > > into the cloud.. because the feed it from PHX2 to the cloud is
> > > expensive.]
> > >
> > > 2. Packaging of elasticsearch was a mess. At the time we had rules
> > > that all packages needed to be packaged in Fedora and follow Fedora
> > > packaging rules. [This one has been relaxed.]
> > >
> > > 3. Running of elasticsearch was a large service in itself. It doesn't
> > > take care of itself and we would need one or more people who know it
> > > well to keep it running. [This goes down the ladder.. the logstash
> > > backends are also full services.. ] Most of that was written in Java
> > > which no one on the team at the time had good experiences with.
> > >
> > > 4. A kibana/elasticsearch query expert. Just like any database, most
> > > of the queries you can make are the worse kind. They will take a lot
> > > more CPU/memory/time than they should making just grepping for data
> > > faster.
> > >
> > > However that is 3-5 years ago.. so a lot has changed since then.
> > >
> > >
> > >> I think that Elasticsearch offers quite a few advantages :
> > >>   - Powerful Query language
> > >>   - Python bindings
> > >>   - Javascript bindings
> > >>   - Can be deployed in our infrastructure or used as a service
> > >>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> > >>
> > >> So what is the general feeling about using Elasticsearch in our
> > >> infrastructure ? Should we look at deploying a cluster in our infra /
> > >> Should we 

Re: Future of fedora-packages

[Matthew Miller]

On Wed, Feb 27, 2019 at 01:04:57PM -0800, Jim Perrin wrote:
> How much heresy is involved in us using Amazon's elasticsearch service
> for this, so that we don't have yet-another-thing to maintain?

I don't think this is heresy at all. Hosted open source services are great,
and the Amazon web services team has been very generous to the project.


-- 
Matthew Miller

Fedora Project Leader
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Jim Perrin]

On 2/27/19 11:29 PM, Clement Verna wrote:
> On Wed, 27 Feb 2019 at 22:11, Jim Perrin  wrote:
>>
>> How much heresy is involved in us using Amazon's elasticsearch service
>> for this, so that we don't have yet-another-thing to maintain?
> 
> If we want to proceed with elasticsearch I think that this is indeed a
> good way forward. What would it take to use our current AWS account to
> setup a test cluster ?
> 

A bit of time writing up some lambda rules to restrict aws resources to
fas groups so that we can provide appropriate boundaries based on group
(
https://aws.amazon.com/blogs/security/how-to-automatically-tag-amazon-ec2-resources-in-response-to-api-events/
)

A fas group for the folks doing the work, and then someone to do it,
roughly speaking.



___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Kevin Fenzi]

On 2/26/19 11:38 AM, Clement Verna wrote:
> Hi all,
> 
> fedora-packages [0] code base is showing its age. The code base and
> the technology stack  (Turbogears2 [1] web framework and the Moksha
> [2] middleware) is currently not ready for Python3 and I am not
> planning to do the work required to make it Python3 compatible, so the
> application will stop working when Fedora 29 is EOL.
> 
> In order to keep the service running, I have started a Proof Of
> Concept (fedora-search [3]) to replace the backend of the application.
> Fedora-search would be a REST API service offering full test search
> API. Such a service would then be available for other application to
> use, fedora-packages would then become a frontend only application
> using the service provided by fedora-search.
> 
> While the POC shows that this is a viable solution, I don't think that
> we should be proceeding that way, for the simple reason that this add
> yet another code base to maintain, I think we should use this
> opportunity to consider using Elasticsearch instead of maintaining our
> own "search engine".
> 
> I think that Elasticsearch offers quite a few advantages :
>   - Powerful Query language
>   - Python bindings
>   - Javascript bindings
>   - Can be deployed in our infrastructure or used as a service
>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> 
> So what is the general feeling about using Elasticsearch in our
> infrastructure ? Should we look at deploying a cluster in our infra /
> Should we approach the Council to see if we can get founding to have
> this service hosted by Elastic ?

Well, I am not sure this could really work, but I don't know enough
about ES to say. What would the frontend look like?

I think users currently use packages to look at information about
packages in a structured way. A "homepage" for the package if you will.
I am sure ES could pull the rpm data and let people search it, but could
it show a nice page with all the various types of information about a
particular package? Or would it just say "here's the first page of your
search results" and let users have to try and craft a search result that
will find the info they are looking for? I think that would be a pretty
big step back.

I agree with the points smooge brought up eariler about us running it
(rpm difficult, etc), but now we do have openshift and it does ship with
ES and might be easy to add to an app. But I am not sure if that helps
if we still need a app to use it, xiapan isn't the issue here right?

Finally, I wonder if we could look at what other distros do? Could we
work with say debian on theirs and add ability to theme it and handle
rpm vs deb data?

kevin



signature.asc
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Neal Gompa]

On Sat, Mar 2, 2019 at 3:31 PM Kevin Fenzi  wrote:
>
> On 2/26/19 11:38 AM, Clement Verna wrote:
> > Hi all,
> >
> > fedora-packages [0] code base is showing its age. The code base and
> > the technology stack  (Turbogears2 [1] web framework and the Moksha
> > [2] middleware) is currently not ready for Python3 and I am not
> > planning to do the work required to make it Python3 compatible, so the
> > application will stop working when Fedora 29 is EOL.
> >
> > In order to keep the service running, I have started a Proof Of
> > Concept (fedora-search [3]) to replace the backend of the application.
> > Fedora-search would be a REST API service offering full test search
> > API. Such a service would then be available for other application to
> > use, fedora-packages would then become a frontend only application
> > using the service provided by fedora-search.
> >
> > While the POC shows that this is a viable solution, I don't think that
> > we should be proceeding that way, for the simple reason that this add
> > yet another code base to maintain, I think we should use this
> > opportunity to consider using Elasticsearch instead of maintaining our
> > own "search engine".
> >
> > I think that Elasticsearch offers quite a few advantages :
> >   - Powerful Query language
> >   - Python bindings
> >   - Javascript bindings
> >   - Can be deployed in our infrastructure or used as a service
> >   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> >
> > So what is the general feeling about using Elasticsearch in our
> > infrastructure ? Should we look at deploying a cluster in our infra /
> > Should we approach the Council to see if we can get founding to have
> > this service hosted by Elastic ?
>
> Well, I am not sure this could really work, but I don't know enough
> about ES to say. What would the frontend look like?
>
> I think users currently use packages to look at information about
> packages in a structured way. A "homepage" for the package if you will.
> I am sure ES could pull the rpm data and let people search it, but could
> it show a nice page with all the various types of information about a
> particular package? Or would it just say "here's the first page of your
> search results" and let users have to try and craft a search result that
> will find the info they are looking for? I think that would be a pretty
> big step back.
>
> I agree with the points smooge brought up eariler about us running it
> (rpm difficult, etc), but now we do have openshift and it does ship with
> ES and might be easy to add to an app. But I am not sure if that helps
> if we still need a app to use it, xiapan isn't the issue here right?
>
> Finally, I wonder if we could look at what other distros do? Could we
> work with say debian on theirs and add ability to theme it and handle
> rpm vs deb data?
>

openSUSE has a package search system[1] that can leverage AppStream
data too, but I'm not sure how its backend works. I think it currently
leverages OBS specific APIs.

As for Debian's package search system[2], it looks like a nightmare... :(

Mageia's App DB[3] is similar to openSUSE's system, but I can't quite
tell if it's just only Mageia branded or relies on Mageia-specific
things. It seems to have some aspects of Bodhi in it too...

As insane as it is, it looks like our existing system is the most
distro agnostic approach I've seen! It's relatively easy to use it for
any arbitrary set of repositories, and the branding is easy enough to
swap out. It's mainly a matter of disabling the code to talk to Bodhi
and simplify the package view page to remove the Fedora-specific bodhi
and fedmsg integration stuff.

If it had the ability to handle Debian packages, it'd probably be a
decent replacement for a number of these. But then again, our tool
doesn't necessarily need to be aware of the difference between RPMs
and DEBs, because it uses mdapi, and an alternative implementation for
Debian repos would be sufficient to make it work.

[1]: https://github.com/openSUSE/software-o-o
[2]: https://salsa.debian.org/webmaster-team/packages
[3]: https://github.com/agallou/mageia-app-db

-- 
真実はいつも一つ！/ Always, there's only one truth!
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

On Sat, 2 Mar 2019 at 21:36, Kevin Fenzi  wrote:
>
> On 2/26/19 11:38 AM, Clement Verna wrote:
> > Hi all,
> >
> > fedora-packages [0] code base is showing its age. The code base and
> > the technology stack  (Turbogears2 [1] web framework and the Moksha
> > [2] middleware) is currently not ready for Python3 and I am not
> > planning to do the work required to make it Python3 compatible, so the
> > application will stop working when Fedora 29 is EOL.
> >
> > In order to keep the service running, I have started a Proof Of
> > Concept (fedora-search [3]) to replace the backend of the application.
> > Fedora-search would be a REST API service offering full test search
> > API. Such a service would then be available for other application to
> > use, fedora-packages would then become a frontend only application
> > using the service provided by fedora-search.
> >
> > While the POC shows that this is a viable solution, I don't think that
> > we should be proceeding that way, for the simple reason that this add
> > yet another code base to maintain, I think we should use this
> > opportunity to consider using Elasticsearch instead of maintaining our
> > own "search engine".
> >
> > I think that Elasticsearch offers quite a few advantages :
> >   - Powerful Query language
> >   - Python bindings
> >   - Javascript bindings
> >   - Can be deployed in our infrastructure or used as a service
> >   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> >
> > So what is the general feeling about using Elasticsearch in our
> > infrastructure ? Should we look at deploying a cluster in our infra /
> > Should we approach the Council to see if we can get founding to have
> > this service hosted by Elastic ?
>
> Well, I am not sure this could really work, but I don't know enough
> about ES to say. What would the frontend look like?

The frontend will be an independent application that we will need to
develop or as suggested by Ryan we may explore using src.fp.o. Having
a separated backend for search capabilities makes a lot of sense to me
since that makes this service easy to reuse by other apps and let
people write their custom client if they wish to do so.

>
> I think users currently use packages to look at information about
> packages in a structured way. A "homepage" for the package if you will.
> I am sure ES could pull the rpm data and let people search it, but could
> it show a nice page with all the various types of information about a
> particular package? Or would it just say "here's the first page of your
> search results" and let users have to try and craft a search result that
> will find the info they are looking for? I think that would be a pretty
> big step back.
>
> I agree with the points smooge brought up eariler about us running it
> (rpm difficult, etc), but now we do have openshift and it does ship with
> ES and might be easy to add to an app. But I am not sure if that helps
> if we still need a app to use it, xiapan isn't the issue here right?

With the idea to have a separate search backend with HTTP API, xapian
is a problem since we have to write the HTTP wrapper around xapian's
API to make it available to all, there is xapiand
(https://kronuz.io/Xapiand/) but the project seems fairly recent and
not yet mature. The other benefits of Elasticsearch are a maintained
Python client, a maintained Javascript client, better documentation,
largely adopted solution (easier to get people interested in).

>
> Finally, I wonder if we could look at what other distros do? Could we
> work with say debian on theirs and add ability to theme it and handle
> rpm vs deb data?

Yes +1 that's sounds like a good idea.

>
> kevin
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Kevin Fenzi]

On 3/3/19 11:28 PM, Clement Verna wrote:

> The frontend will be an independent application that we will need to
> develop or as suggested by Ryan we may explore using src.fp.o. Having
> a separated backend for search capabilities makes a lot of sense to me
> since that makes this service easy to reuse by other apps and let
> people write their custom client if they wish to do so.

So, really this is about replacing xiapan with ES, right?
Since we will still need a app to interface with those results..

> With the idea to have a separate search backend with HTTP API, xapian
> is a problem since we have to write the HTTP wrapper around xapian's
> API to make it available to all, there is xapiand
> (https://kronuz.io/Xapiand/) but the project seems fairly recent and
> not yet mature. The other benefits of Elasticsearch are a maintained
> Python client, a maintained Javascript client, better documentation,
> largely adopted solution (easier to get people interested in).

Yeah, all good points.

kevin



signature.asc
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

On Mon, 4 Mar 2019 at 19:21, Kevin Fenzi  wrote:
>
> On 3/3/19 11:28 PM, Clement Verna wrote:
>
> > The frontend will be an independent application that we will need to
> > develop or as suggested by Ryan we may explore using src.fp.o. Having
> > a separated backend for search capabilities makes a lot of sense to me
> > since that makes this service easy to reuse by other apps and let
> > people write their custom client if they wish to do so.
>
> So, really this is about replacing xiapan with ES, right?
> Since we will still need a app to interface with those results..

Yes this is the first step for me, once we replace the backend we can
get creative on how we want to replace the frontend :-)

>
> > With the idea to have a separate search backend with HTTP API, xapian
> > is a problem since we have to write the HTTP wrapper around xapian's
> > API to make it available to all, there is xapiand
> > (https://kronuz.io/Xapiand/) but the project seems fairly recent and
> > not yet mature. The other benefits of Elasticsearch are a maintained
> > Python client, a maintained Javascript client, better documentation,
> > largely adopted solution (easier to get people interested in).
>
> Yeah, all good points.
>
> kevin
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Randy Barlow]

On Wed, 2019-02-27 at 10:48 +0100, Pierre-Yves Chibon wrote:
> We look at Elasticsearch at one point (I believe Aurélien was even
> looking at
> setting up an instance in our cloud) but it came down to two aspects:
> - setting it up is another application to maintain
> - an application we have no expertise on
> 
> On the other side, fedora-search is:
> - setting up another application to maintain
> - an application we have expertise on

Well, fedora-search is also:

- an application we have to create.
- an application we have to maintain the code and the deployment,
  where elasticsearch is just maintaining the deployment.


signature.asc
Description: This is a digitally signed message part
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Randy Barlow]

On Tue, 2019-02-26 at 20:38 +0100, Clement Verna wrote:
> Should we approach the Council to see if we can get founding to have
> this service hosted by Elastic ?

If we use the service hosted by Elastic that's a nice way to support
development of an open source project with some cold hard cash.


signature.asc
Description: This is a digitally signed message part
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

So the fun fact is that we are already running elasticsearch with
kibana and logstash [0] in our Openshift clusters. So maybe we can
leverage on how this is deployed to run a dedicated instance for
fedora-packages.

[0] - https://kibana.app.os.fedoraproject.org/

On Fri, 8 Mar 2019 at 17:42, Randy Barlow  wrote:
>
> On Tue, 2019-02-26 at 20:38 +0100, Clement Verna wrote:
> > Should we approach the Council to see if we can get founding to have
> > this service hosted by Elastic ?
>
> If we use the service hosted by Elastic that's a nice way to support
> development of an open source project with some cold hard cash.
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Kevin Fenzi]

On 3/27/19 3:50 AM, Clement Verna wrote:
> So the fun fact is that we are already running elasticsearch with
> kibana and logstash [0] in our Openshift clusters. So maybe we can
> leverage on how this is deployed to run a dedicated instance for
> fedora-packages.

Yeah.

However, it's pretty tailored to logs of pods. It would likely be easier
to start with a dedicated elastic pod for searching packages, but then
you would need to teach it how to gather the info and write a frontend
for it. :(

kevin



signature.asc
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

On Thu, 28 Mar 2019 at 16:27, Kevin Fenzi  wrote:
>
> On 3/27/19 3:50 AM, Clement Verna wrote:
> > So the fun fact is that we are already running elasticsearch with
> > kibana and logstash [0] in our Openshift clusters. So maybe we can
> > leverage on how this is deployed to run a dedicated instance for
> > fedora-packages.
>
> Yeah.
>
> However, it's pretty tailored to logs of pods. It would likely be easier
> to start with a dedicated elastic pod for searching packages, but then
> you would need to teach it how to gather the info and write a frontend
> for it. :(

Yes, it does not look like we can easily reuse this. I think our best
chance is to work on making src.fp.o the new central place for
packages, but we need to improve the search functionality of Pagure. I
personally don't think we will have enough time to work on making a
fedora-packages 2.0 except if someone else is interested in working on
it.

>
> kevin
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Kevin Fenzi]

On 3/28/19 8:54 AM, Clement Verna wrote:
> On Thu, 28 Mar 2019 at 16:27, Kevin Fenzi  wrote:
>>
>> On 3/27/19 3:50 AM, Clement Verna wrote:
>>> So the fun fact is that we are already running elasticsearch with
>>> kibana and logstash [0] in our Openshift clusters. So maybe we can
>>> leverage on how this is deployed to run a dedicated instance for
>>> fedora-packages.
>>
>> Yeah.
>>
>> However, it's pretty tailored to logs of pods. It would likely be easier
>> to start with a dedicated elastic pod for searching packages, but then
>> you would need to teach it how to gather the info and write a frontend
>> for it. :(
> 
> Yes, it does not look like we can easily reuse this. I think our best
> chance is to work on making src.fp.o the new central place for
> packages, but we need to improve the search functionality of Pagure. I
> personally don't think we will have enough time to work on making a
> fedora-packages 2.0 except if someone else is interested in working on
> it.

Do you think this could be a good project for outreachy or the like if
you have time to mentor? Or is it too big for that kind of timeframe?

kevin




signature.asc
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

On Thu, 28 Mar 2019 at 23:22, Kevin Fenzi  wrote:
>
> On 3/28/19 8:54 AM, Clement Verna wrote:
> > On Thu, 28 Mar 2019 at 16:27, Kevin Fenzi  wrote:
> >>
> >> On 3/27/19 3:50 AM, Clement Verna wrote:
> >>> So the fun fact is that we are already running elasticsearch with
> >>> kibana and logstash [0] in our Openshift clusters. So maybe we can
> >>> leverage on how this is deployed to run a dedicated instance for
> >>> fedora-packages.
> >>
> >> Yeah.
> >>
> >> However, it's pretty tailored to logs of pods. It would likely be easier
> >> to start with a dedicated elastic pod for searching packages, but then
> >> you would need to teach it how to gather the info and write a frontend
> >> for it. :(
> >
> > Yes, it does not look like we can easily reuse this. I think our best
> > chance is to work on making src.fp.o the new central place for
> > packages, but we need to improve the search functionality of Pagure. I
> > personally don't think we will have enough time to work on making a
> > fedora-packages 2.0 except if someone else is interested in working on
> > it.
>
> Do you think this could be a good project for outreachy or the like if
> you have time to mentor? Or is it too big for that kind of timeframe?

Yes that might be an option, it also depends what we want to do. The
existing code base is so coupled to Turbogears2 and moksha so porting
it to another web framework looks like you would need to rewrite most
of it anyway.


>
> kevin
>
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org