Re: Sling to Solr integration using Sling Content Distribution

[Tommaso Teofili]

thanks Dirk, I'm going to review the PR in the next days.

Regards,
Tommaso

Il giorno mer 17 gen 2018 alle ore 14:58 Dirk Rudolph <
dirk.rudo...@netcentric.biz> ha scritto:

> Hey.
>
> I opened a PR for SLING-7364, the missing piece making SCD compatible with
> solr out of the box:
>
> https://github.com/apache/sling-org-apache-sling-distribution-core/pull/6
> <https://github.com/apache/sling-org-apache-sling-distribution-core/pull/6
> >
>
> Beside of that I created an API to create solr documents from resources
> for serialisation here:
>
>
> https://github.com/Buuhuu/sling-content-distribution-solr/tree/master/sling-content-distribution-solr-bundle/src/main/java/com/github/buuhuu/sling/distribution/solr/documents
> <
> https://github.com/Buuhuu/sling-content-distribution-solr/tree/master/sling-content-distribution-solr-bundle/src/main/java/com/github/buuhuu/sling/distribution/solr/documents
> >
>
> With a sample implementation using Adobes AEM Sample “we retail” here:
>
>
> https://github.com/Buuhuu/sling-content-distribution-solr/tree/master/sling-content-distribution-solr-weretail
> <
> https://github.com/Buuhuu/sling-content-distribution-solr/tree/master/sling-content-distribution-solr-weretail
> >
>
> Let me know what you think.
>
> /Dirk
>
> > On 11 Jan 2018, at 15:38, Dirk Rudolph <dirk.rudo...@netcentric.biz>
> wrote:
> >
> > Many thanks for the positive Feedback.
> >
> > The last thing missing to make it work ootb with solr is SLING-7364. I
> added a some thoughts to it. Option 1) is instead of sending the package
> info as binary header prepended to the request’s body, sending them as http
> headers. As the overall size of http headers is limited this would require
> SLING-3967 to be implemented. Another feasible and probably the more simple
> solutions for now is 3) omitting the package info if the agent is
> configured to so. Wdyt?
> >
> > I will keep working on defining an API for the documents to export of
> solr. From my experience with AEM the following approach comes to my mind:
> >
> > For each Resource:
> >
> > 1. Adapt to SolrFieldsIterator, iterating over SolrField (name, value)
> > 2. If not null, add each SolrField to the document and done
> > 3. If null, adapt to SolrFields, a marker interface that can be used
> together with any (existing) model, that has getters annotated with
> @SolrField.
> > 4. If not null add each @SolrField getter to the document and done
> > 5. If null, log and skip
> >
> > About step 1) Thinking about cq:Page or dam:Asset in AEM, or anything
> Resource that should be exported as an aggregate of child resources. The
> SolrFieldsIterator implementation would traverse and itself adapt each
> resource to SolrFields to do the same as step 3).
> >
> > This would be done as a service so it could be replaced by any other
> implementation, like a simple property=>field mapping.
> >
> > Thoughts?
> >
> >> On 10 Jan 2018, at 10:59, Tommaso Teofili <tommaso.teof...@gmail.com>
> wrote:
> >>
> >> +1
> >>
> >> thanks a lot Dirk for your contributions (4 PRs so far!). This sounds
> like
> >> a very interesting use case for Sling and SCD.
> >>
> >> Regards,
> >> Tommaso
> >>
> >>
> >>
> >> Il giorno mar 9 gen 2018 alle ore 02:22 Daniel Klco <
> daniel.k...@gmail.com>
> >> ha scritto:
> >>
> >>> This is a great idea! I could see this as a use case for a number of
> >>> different integrations and an example would be very helpful for anyone
> >>> looking to do this.
> >>>
> >>> On Mon, Jan 8, 2018 at 1:44 PM, Stefan Seifert <sseif...@pro-vision.de
> >
> >>> wrote:
> >>>
> >>>> hello dirk.
> >>>>
> >>>> i think such a feature would be very useful. integration of an
> external
> >>>> search engine on a "higher level" than the oak-level search
> integration
> >>> is
> >>>> a common use case in our projects as well. it's important to be quite
> >>>> flexible what is indexed and what not and how it's indexed - for best
> >>> match
> >>>> of the business requirements and the special features of the targeted
> >>>> search engine.
> >>>>
> >>>> stefan
> >>>>
> >>>>> -Original Message-
> >>>>> From: Dirk Rudolph [mailto:dirk.rudo...@netcentric.biz]
> >>>>> Sent: Monday, January 8, 201

Re: Sling to Solr integration using Sling Content Distribution

[Dirk Rudolph]

Hey. 

I opened a PR for SLING-7364, the missing piece making SCD compatible with solr 
out of the box:

https://github.com/apache/sling-org-apache-sling-distribution-core/pull/6 
<https://github.com/apache/sling-org-apache-sling-distribution-core/pull/6> 

Beside of that I created an API to create solr documents from resources for 
serialisation here:

https://github.com/Buuhuu/sling-content-distribution-solr/tree/master/sling-content-distribution-solr-bundle/src/main/java/com/github/buuhuu/sling/distribution/solr/documents
 
<https://github.com/Buuhuu/sling-content-distribution-solr/tree/master/sling-content-distribution-solr-bundle/src/main/java/com/github/buuhuu/sling/distribution/solr/documents>

With a sample implementation using Adobes AEM Sample “we retail” here:

https://github.com/Buuhuu/sling-content-distribution-solr/tree/master/sling-content-distribution-solr-weretail
 
<https://github.com/Buuhuu/sling-content-distribution-solr/tree/master/sling-content-distribution-solr-weretail>

Let me know what you think.

/Dirk

> On 11 Jan 2018, at 15:38, Dirk Rudolph <dirk.rudo...@netcentric.biz> wrote:
> 
> Many thanks for the positive Feedback. 
> 
> The last thing missing to make it work ootb with solr is SLING-7364. I added 
> a some thoughts to it. Option 1) is instead of sending the package info as 
> binary header prepended to the request’s body, sending them as http headers. 
> As the overall size of http headers is limited this would require SLING-3967 
> to be implemented. Another feasible and probably the more simple solutions 
> for now is 3) omitting the package info if the agent is configured to so. 
> Wdyt?
> 
> I will keep working on defining an API for the documents to export of solr. 
> From my experience with AEM the following approach comes to my mind:
> 
> For each Resource:
> 
> 1. Adapt to SolrFieldsIterator, iterating over SolrField (name, value)
> 2. If not null, add each SolrField to the document and done
> 3. If null, adapt to SolrFields, a marker interface that can be used together 
> with any (existing) model, that has getters annotated with @SolrField. 
> 4. If not null add each @SolrField getter to the document and done
> 5. If null, log and skip
> 
> About step 1) Thinking about cq:Page or dam:Asset in AEM, or anything 
> Resource that should be exported as an aggregate of child resources. The 
> SolrFieldsIterator implementation would traverse and itself adapt each 
> resource to SolrFields to do the same as step 3).
> 
> This would be done as a service so it could be replaced by any other 
> implementation, like a simple property=>field mapping. 
> 
> Thoughts?
> 
>> On 10 Jan 2018, at 10:59, Tommaso Teofili <tommaso.teof...@gmail.com> wrote:
>> 
>> +1
>> 
>> thanks a lot Dirk for your contributions (4 PRs so far!). This sounds like
>> a very interesting use case for Sling and SCD.
>> 
>> Regards,
>> Tommaso
>> 
>> 
>> 
>> Il giorno mar 9 gen 2018 alle ore 02:22 Daniel Klco <daniel.k...@gmail.com>
>> ha scritto:
>> 
>>> This is a great idea! I could see this as a use case for a number of
>>> different integrations and an example would be very helpful for anyone
>>> looking to do this.
>>> 
>>> On Mon, Jan 8, 2018 at 1:44 PM, Stefan Seifert <sseif...@pro-vision.de>
>>> wrote:
>>> 
>>>> hello dirk.
>>>> 
>>>> i think such a feature would be very useful. integration of an external
>>>> search engine on a "higher level" than the oak-level search integration
>>> is
>>>> a common use case in our projects as well. it's important to be quite
>>>> flexible what is indexed and what not and how it's indexed - for best
>>> match
>>>> of the business requirements and the special features of the targeted
>>>> search engine.
>>>> 
>>>> stefan
>>>> 
>>>>> -Original Message-
>>>>> From: Dirk Rudolph [mailto:dirk.rudo...@netcentric.biz]
>>>>> Sent: Monday, January 8, 2018 4:45 PM
>>>>> To: dev@sling.apache.org
>>>>> Subject: Sling to Solr integration using Sling Content Distribution
>>>>> 
>>>>> Hi devs,
>>>>> 
>>>>> Recently I was evaluating if it is easily possible to integrate Sling
>>> into
>>>>> Solr using Sling Content Distributions and made some great progress
>>> here:
>>>>> 
>>>>> https://github.com/Buuhuu/sling-content-distribution-solr
>>>>> <https://github.com/Buuhuu/sling-content-distribution-solr>
&

Re: Sling to Solr integration using Sling Content Distribution

[Dirk Rudolph]

Many thanks for the positive Feedback. 

The last thing missing to make it work ootb with solr is SLING-7364. I added a 
some thoughts to it. Option 1) is instead of sending the package info as binary 
header prepended to the request’s body, sending them as http headers. As the 
overall size of http headers is limited this would require SLING-3967 to be 
implemented. Another feasible and probably the more simple solutions for now is 
3) omitting the package info if the agent is configured to so. Wdyt?

I will keep working on defining an API for the documents to export of solr. 
From my experience with AEM the following approach comes to my mind:

For each Resource:

1. Adapt to SolrFieldsIterator, iterating over SolrField (name, value)
2. If not null, add each SolrField to the document and done
3. If null, adapt to SolrFields, a marker interface that can be used together 
with any (existing) model, that has getters annotated with @SolrField. 
4. If not null add each @SolrField getter to the document and done
5. If null, log and skip

About step 1) Thinking about cq:Page or dam:Asset in AEM, or anything Resource 
that should be exported as an aggregate of child resources. The 
SolrFieldsIterator implementation would traverse and itself adapt each resource 
to SolrFields to do the same as step 3).

This would be done as a service so it could be replaced by any other 
implementation, like a simple property=>field mapping. 

Thoughts?

> On 10 Jan 2018, at 10:59, Tommaso Teofili <tommaso.teof...@gmail.com> wrote:
> 
> +1
> 
> thanks a lot Dirk for your contributions (4 PRs so far!). This sounds like
> a very interesting use case for Sling and SCD.
> 
> Regards,
> Tommaso
> 
> 
> 
> Il giorno mar 9 gen 2018 alle ore 02:22 Daniel Klco <daniel.k...@gmail.com>
> ha scritto:
> 
>> This is a great idea! I could see this as a use case for a number of
>> different integrations and an example would be very helpful for anyone
>> looking to do this.
>> 
>> On Mon, Jan 8, 2018 at 1:44 PM, Stefan Seifert <sseif...@pro-vision.de>
>> wrote:
>> 
>>> hello dirk.
>>> 
>>> i think such a feature would be very useful. integration of an external
>>> search engine on a "higher level" than the oak-level search integration
>> is
>>> a common use case in our projects as well. it's important to be quite
>>> flexible what is indexed and what not and how it's indexed - for best
>> match
>>> of the business requirements and the special features of the targeted
>>> search engine.
>>> 
>>> stefan
>>> 
>>>> -Original Message-
>>>> From: Dirk Rudolph [mailto:dirk.rudo...@netcentric.biz]
>>>> Sent: Monday, January 8, 2018 4:45 PM
>>>> To: dev@sling.apache.org
>>>> Subject: Sling to Solr integration using Sling Content Distribution
>>>> 
>>>> Hi devs,
>>>> 
>>>> Recently I was evaluating if it is easily possible to integrate Sling
>> into
>>>> Solr using Sling Content Distributions and made some great progress
>> here:
>>>> 
>>>> https://github.com/Buuhuu/sling-content-distribution-solr
>>>> <https://github.com/Buuhuu/sling-content-distribution-solr>
>>>> 
>>>> The repository explains the goal, the why and who also giving
>> instructions
>>>> on how to use it. To sum it up a bit:
>>>> 
>>>> - SCD’s features map perfectly fine to the requirements when integrating
>>>> into solr
>>>> - It enables us to ingest content on business perspective (not technical
>>>> perspective as with Oak’s internal indexes)
>>>> - It’s flexible (thanks for that already :)
>>>> 
>>>> Though there are some things that require a bit of attention to make it
>>>> work out of the box. I opened a couple of issues for that:
>>>> 
>>>> SLING-7357
>>>> SLING-7358
>>>> SLING-7359
>>>> SLING-7360
>>>> SLING-7364
>>>> 
>>>> And made proposals accordingly (not for the last one as I want to
>> discuss
>>>> that first)
>>>> 
>>>> So my general question is:
>>>> 
>>>> Is integrating Sling into Solr (or potentially any other kind of system
>>>> that offers APIs to do so) a valid and envisaged use-case for Sling
>>> Content
>>>> Distribution? And if so would it make sense to implement a module for
>> lets
>>>> say Solr as example directly in Sling?
>>>> 
>>>> If so I would volunteer to propose something for that but I think
>> flexibly
>>>> integrating Sling as framework for any kind of content driven web
>>>> applications into Solr as enterprise search application would be a nice
>>>> feature to offer.
>>>> 
>>>> Thanks for any kind of feedback,
>>>> 
>>>> /Dirk
>>> 
>> 

Re: Sling to Solr integration using Sling Content Distribution

[Tommaso Teofili]

+1

thanks a lot Dirk for your contributions (4 PRs so far!). This sounds like
a very interesting use case for Sling and SCD.

Regards,
Tommaso



Il giorno mar 9 gen 2018 alle ore 02:22 Daniel Klco <daniel.k...@gmail.com>
ha scritto:

> This is a great idea! I could see this as a use case for a number of
> different integrations and an example would be very helpful for anyone
> looking to do this.
>
> On Mon, Jan 8, 2018 at 1:44 PM, Stefan Seifert <sseif...@pro-vision.de>
> wrote:
>
> > hello dirk.
> >
> > i think such a feature would be very useful. integration of an external
> > search engine on a "higher level" than the oak-level search integration
> is
> > a common use case in our projects as well. it's important to be quite
> > flexible what is indexed and what not and how it's indexed - for best
> match
> > of the business requirements and the special features of the targeted
> > search engine.
> >
> > stefan
> >
> > >-Original Message-
> > >From: Dirk Rudolph [mailto:dirk.rudo...@netcentric.biz]
> > >Sent: Monday, January 8, 2018 4:45 PM
> > >To: dev@sling.apache.org
> > >Subject: Sling to Solr integration using Sling Content Distribution
> > >
> > >Hi devs,
> > >
> > >Recently I was evaluating if it is easily possible to integrate Sling
> into
> > >Solr using Sling Content Distributions and made some great progress
> here:
> > >
> > >https://github.com/Buuhuu/sling-content-distribution-solr
> > ><https://github.com/Buuhuu/sling-content-distribution-solr>
> > >
> > >The repository explains the goal, the why and who also giving
> instructions
> > >on how to use it. To sum it up a bit:
> > >
> > >- SCD’s features map perfectly fine to the requirements when integrating
> > >into solr
> > >- It enables us to ingest content on business perspective (not technical
> > >perspective as with Oak’s internal indexes)
> > >- It’s flexible (thanks for that already :)
> > >
> > >Though there are some things that require a bit of attention to make it
> > >work out of the box. I opened a couple of issues for that:
> > >
> > >SLING-7357
> > >SLING-7358
> > >SLING-7359
> > >SLING-7360
> > >SLING-7364
> > >
> > >And made proposals accordingly (not for the last one as I want to
> discuss
> > >that first)
> > >
> > >So my general question is:
> > >
> > >Is integrating Sling into Solr (or potentially any other kind of system
> > >that offers APIs to do so) a valid and envisaged use-case for Sling
> > Content
> > >Distribution? And if so would it make sense to implement a module for
> lets
> > >say Solr as example directly in Sling?
> > >
> > >If so I would volunteer to propose something for that but I think
> flexibly
> > >integrating Sling as framework for any kind of content driven web
> > >applications into Solr as enterprise search application would be a nice
> > >feature to offer.
> > >
> > >Thanks for any kind of feedback,
> > >
> > >/Dirk
> >
>

Re: Sling to Solr integration using Sling Content Distribution

[Daniel Klco]

This is a great idea! I could see this as a use case for a number of
different integrations and an example would be very helpful for anyone
looking to do this.

On Mon, Jan 8, 2018 at 1:44 PM, Stefan Seifert <sseif...@pro-vision.de>
wrote:

> hello dirk.
>
> i think such a feature would be very useful. integration of an external
> search engine on a "higher level" than the oak-level search integration is
> a common use case in our projects as well. it's important to be quite
> flexible what is indexed and what not and how it's indexed - for best match
> of the business requirements and the special features of the targeted
> search engine.
>
> stefan
>
> >-Original Message-
> >From: Dirk Rudolph [mailto:dirk.rudo...@netcentric.biz]
> >Sent: Monday, January 8, 2018 4:45 PM
> >To: dev@sling.apache.org
> >Subject: Sling to Solr integration using Sling Content Distribution
> >
> >Hi devs,
> >
> >Recently I was evaluating if it is easily possible to integrate Sling into
> >Solr using Sling Content Distributions and made some great progress here:
> >
> >https://github.com/Buuhuu/sling-content-distribution-solr
> ><https://github.com/Buuhuu/sling-content-distribution-solr>
> >
> >The repository explains the goal, the why and who also giving instructions
> >on how to use it. To sum it up a bit:
> >
> >- SCD’s features map perfectly fine to the requirements when integrating
> >into solr
> >- It enables us to ingest content on business perspective (not technical
> >perspective as with Oak’s internal indexes)
> >- It’s flexible (thanks for that already :)
> >
> >Though there are some things that require a bit of attention to make it
> >work out of the box. I opened a couple of issues for that:
> >
> >SLING-7357
> >SLING-7358
> >SLING-7359
> >SLING-7360
> >SLING-7364
> >
> >And made proposals accordingly (not for the last one as I want to discuss
> >that first)
> >
> >So my general question is:
> >
> >Is integrating Sling into Solr (or potentially any other kind of system
> >that offers APIs to do so) a valid and envisaged use-case for Sling
> Content
> >Distribution? And if so would it make sense to implement a module for lets
> >say Solr as example directly in Sling?
> >
> >If so I would volunteer to propose something for that but I think flexibly
> >integrating Sling as framework for any kind of content driven web
> >applications into Solr as enterprise search application would be a nice
> >feature to offer.
> >
> >Thanks for any kind of feedback,
> >
> >/Dirk
>

RE: Sling to Solr integration using Sling Content Distribution

[Stefan Seifert]

hello dirk.

i think such a feature would be very useful. integration of an external search 
engine on a "higher level" than the oak-level search integration is a common 
use case in our projects as well. it's important to be quite flexible what is 
indexed and what not and how it's indexed - for best match of the business 
requirements and the special features of the targeted search engine.

stefan

>-Original Message-
>From: Dirk Rudolph [mailto:dirk.rudo...@netcentric.biz]
>Sent: Monday, January 8, 2018 4:45 PM
>To: dev@sling.apache.org
>Subject: Sling to Solr integration using Sling Content Distribution
>
>Hi devs,
>
>Recently I was evaluating if it is easily possible to integrate Sling into
>Solr using Sling Content Distributions and made some great progress here:
>
>https://github.com/Buuhuu/sling-content-distribution-solr
><https://github.com/Buuhuu/sling-content-distribution-solr>
>
>The repository explains the goal, the why and who also giving instructions
>on how to use it. To sum it up a bit:
>
>- SCD’s features map perfectly fine to the requirements when integrating
>into solr
>- It enables us to ingest content on business perspective (not technical
>perspective as with Oak’s internal indexes)
>- It’s flexible (thanks for that already :)
>
>Though there are some things that require a bit of attention to make it
>work out of the box. I opened a couple of issues for that:
>
>SLING-7357
>SLING-7358
>SLING-7359
>SLING-7360
>SLING-7364
>
>And made proposals accordingly (not for the last one as I want to discuss
>that first)
>
>So my general question is:
>
>Is integrating Sling into Solr (or potentially any other kind of system
>that offers APIs to do so) a valid and envisaged use-case for Sling Content
>Distribution? And if so would it make sense to implement a module for lets
>say Solr as example directly in Sling?
>
>If so I would volunteer to propose something for that but I think flexibly
>integrating Sling as framework for any kind of content driven web
>applications into Solr as enterprise search application would be a nice
>feature to offer.
>
>Thanks for any kind of feedback,
>
>/Dirk

Sling to Solr integration using Sling Content Distribution

[Dirk Rudolph]

Hi devs,

Recently I was evaluating if it is easily possible to integrate Sling into Solr 
using Sling Content Distributions and made some great progress here: 

https://github.com/Buuhuu/sling-content-distribution-solr 


The repository explains the goal, the why and who also giving instructions on 
how to use it. To sum it up a bit:

- SCD’s features map perfectly fine to the requirements when integrating into 
solr
- It enables us to ingest content on business perspective (not technical 
perspective as with Oak’s internal indexes)
- It’s flexible (thanks for that already :)

Though there are some things that require a bit of attention to make it work 
out of the box. I opened a couple of issues for that:

SLING-7357
SLING-7358
SLING-7359
SLING-7360
SLING-7364

And made proposals accordingly (not for the last one as I want to discuss that 
first)

So my general question is:

Is integrating Sling into Solr (or potentially any other kind of system that 
offers APIs to do so) a valid and envisaged use-case for Sling Content 
Distribution? And if so would it make sense to implement a module for lets say 
Solr as example directly in Sling? 

If so I would volunteer to propose something for that but I think flexibly 
integrating Sling as framework for any kind of content driven web applications 
into Solr as enterprise search application would be a nice feature to offer. 

Thanks for any kind of feedback,

/Dirk