Re: [Dspace-tech] Solr config parameter interpolation

2013-10-30 Thread Andrea Bollini 

  
  
Hi all,
  I have merged "the safer approach" to restore our previous
  configuration for autocommit as in PR#368 provided by Andrea
  Pascarelli.
  https://github.com/DSpace/DSpace/pull/368
  As we can do better, see links send before by AndreaP or this most
  concise but well done blog article
http://www.opensourceconnections.com/2013/04/25/understanding-solr-soft-commits-and-data-durability/
  
  I highly suggest that others will take a look to the PR#370. I
  will prefer to have this change widely tested so I suggest to get
  it in before RC1 (so to use it during testhaton) or postpone it to
  a 4.1
  https://github.com/DSpace/DSpace/pull/370
  
  Andrea
  
  
  
  Il 29/10/2013 10.58, Pascarelli Luigi Andrea ha scritto:


  
  To be thorough, autoCommit works doing commit every 15sec but do
  that not cause a new searcher to be opened to make changes visible
  (because openSearcher is set to false). The parameter
  autoSoftCommit is setted by default to -1 meaning that no action
  to do. So having openSearcher=false is not the best choice, maybe
  try to change that setting could be not need to do a soft commit
  either the change proposed by me in the previous email. Surely
  have sense to change the configuration to obtain an autoCommit
  every 10sec or maxDocs reach the 1 documents (like the old
  DSpace solrconfig.xml says). Note that this changes would to apply
  to all Solr core used by DSpace (search/statistics/oai).
  For more informations see:
  https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
  https://cwiki.apache.org/confluence/display/solr/Near+Real+Time+Searching
  
  Regards,
  
  Luigi Andrea
  
  Il 29/10/2013 01:37, Pascarelli Luigi
Andrea ha scritto:
  
  

Hi all,
solr supports system property substitution as you can see at http://wiki.apache.org/solr/SolrConfigXml#System_property_substitution
So the two mentioned parameters can be tuned as you like using
autoCommit and autoSoftCommit features.
The strange behaviour noticed by Robin is because the discovery
consumer call an index method that do not call the
programmatically solr commit, which the update-discovery do well
(as noticed by helix).
Imho the https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/discovery/IndexEventConsumer.java#L170
could be modified with calling the "void indexContent(Context
context, DSpaceObject dso, boolean force, boolean commit)"
method and force commit to true (or get this information from
configuration to left the user to use autoSoftCommit in case the
commit property is false)
Hope this helps.

Regards,

Luigi Andrea

Il 18/10/2013 15:57, TAYLOR Robin
  ha scritto:


  Hi,

I suspect, but I am definitely not sure, that no automatic commits will be
done with the config as it is. If I run an 'optimize' using the Solr Admin
client, or restart Tomcat, then the indices get updated. Not a big issue,
I think we maybe just need to set some default values. I'll do a wee bit
more investigation.

Cheers.

 

On 18/10/2013 14:43, "helix84"  wrote:


  
I can confirm I'm seeing the same thing in DSpace 4 and that DSpace 3
had actual numbers there. I join you in your confusion.

Can you observe any ill effects? I didn't notice anything wrong and I
ran "update-discovery-index -f" on ~20k items yesterday.


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

  


-- 
  
  
  
  
  

  

 

Luigi Andrea Pascarelli
   Dipartimento Servizi e Soluzioni per
  l'Amministrazione Universitaria
  Divisione Ricerca
  Via dei Tizii, 6 
00185 Roma, Italy 
  ph. +39 06 59292895
http://www.cineca.it
  
  

  
  




--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk



___
DSpac

Re: [Dspace-tech] Solr config parameter interpolation

2013-10-29 Thread Pascarelli Luigi Andrea 

  
  
To be thorough, autoCommit works doing commit every 15sec but do
that not cause a new searcher to be opened to make changes visible
(because openSearcher is set to false). The parameter autoSoftCommit
is setted by default to -1 meaning that no action to do. So having
openSearcher=false is not the best choice, maybe try to change that
setting could be not need to do a soft commit either the change
proposed by me in the previous email. Surely have sense to change
the configuration to obtain an autoCommit every 10sec or maxDocs
reach the 1 documents (like the old DSpace solrconfig.xml says).
Note that this changes would to apply to all Solr core used by
DSpace (search/statistics/oai).
For more informations see:
https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
https://cwiki.apache.org/confluence/display/solr/Near+Real+Time+Searching

Regards,

Luigi Andrea

Il 29/10/2013 01:37, Pascarelli Luigi
  Andrea ha scritto:


  
  Hi all,
  solr supports system property substitution as you can see at http://wiki.apache.org/solr/SolrConfigXml#System_property_substitution
  So the two mentioned parameters can be tuned as you like using
  autoCommit and autoSoftCommit features.
  The strange behaviour noticed by Robin is because the discovery
  consumer call an index method that do not call the
  programmatically solr commit, which the update-discovery do well
  (as noticed by helix).
  Imho the https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/discovery/IndexEventConsumer.java#L170
  could be modified with calling the "void indexContent(Context
  context, DSpaceObject dso, boolean force, boolean commit)" method
  and force commit to true (or get this information from
  configuration to left the user to use autoSoftCommit in case the
  commit property is false)
  Hope this helps.
  
  Regards,
  
  Luigi Andrea
  
  Il 18/10/2013 15:57, TAYLOR Robin ha
scritto:
  
  
Hi,

I suspect, but I am definitely not sure, that no automatic commits will be
done with the config as it is. If I run an 'optimize' using the Solr Admin
client, or restart Tomcat, then the indices get updated. Not a big issue,
I think we maybe just need to set some default values. I'll do a wee bit
more investigation.

Cheers.

 

On 18/10/2013 14:43, "helix84"  wrote:



  I can confirm I'm seeing the same thing in DSpace 4 and that DSpace 3
had actual numbers there. I join you in your confusion.

Can you observe any ill effects? I didn't notice anything wrong and I
ran "update-discovery-index -f" on ~20k items yesterday.


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette




  
  
  -- 





  

  
   
  
  Luigi

Andrea Pascarelli
 Dipartimento Servizi e Soluzioni per
l'Amministrazione Universitaria
Divisione Ricerca
Via dei Tizii, 6 
  00185 Roma, Italy 
ph. +39 06 59292895
  http://www.cineca.it


  


  
  
  
  
  --
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
  
  
  
  ___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette


-- 
  
  
  
  
  

  

 

Luigi
  Andrea Pascarelli
  
Dipartimento Servizi e Soluzioni per
  l'Amministrazione Universitaria
  Divisione Ricerca
  Via

dei Tizii, 6 
00185 Roma, Italy 
  ph. +39 06 59292895
http://www.cineca.it
  
  

  
  

  

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more a

Re: [Dspace-tech] Solr config parameter interpolation

2013-10-28 Thread Pascarelli Luigi Andrea 

  
  
Hi all,
solr supports system property substitution as you can see at
http://wiki.apache.org/solr/SolrConfigXml#System_property_substitution
So the two mentioned parameters can be tuned as you like using
autoCommit and autoSoftCommit features.
The strange behaviour noticed by Robin is because the discovery
consumer call an index method that do not call the programmatically
solr commit, which the update-discovery do well (as noticed by
helix).
Imho the
https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/discovery/IndexEventConsumer.java#L170
could be modified with calling the "void indexContent(Context
context, DSpaceObject dso, boolean force, boolean commit)" method
and force commit to true (or get this information from configuration
to left the user to use autoSoftCommit in case the commit property
is false)
Hope this helps.

Regards,

Luigi Andrea

Il 18/10/2013 15:57, TAYLOR Robin ha
  scritto:


  Hi,

I suspect, but I am definitely not sure, that no automatic commits will be
done with the config as it is. If I run an 'optimize' using the Solr Admin
client, or restart Tomcat, then the indices get updated. Not a big issue,
I think we maybe just need to set some default values. I'll do a wee bit
more investigation.

Cheers.

 

On 18/10/2013 14:43, "helix84"  wrote:


  
I can confirm I'm seeing the same thing in DSpace 4 and that DSpace 3
had actual numbers there. I join you in your confusion.

Can you observe any ill effects? I didn't notice anything wrong and I
ran "update-discovery-index -f" on ~20k items yesterday.


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

  
  




-- 
  
  
  
  
  

  

 

Luigi
  Andrea Pascarelli
  
Dipartimento Servizi e Soluzioni per
  l'Amministrazione Universitaria
  Divisione Ricerca
  Via

dei Tizii, 6 
00185 Roma, Italy 
  ph. +39 06 59292895
http://www.cineca.it
  
  

  
  

  

--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr config parameter interpolation

2013-10-18 Thread TAYLOR Robin 
Hi,

I suspect, but I am definitely not sure, that no automatic commits will be
done with the config as it is. If I run an 'optimize' using the Solr Admin
client, or restart Tomcat, then the indices get updated. Not a big issue,
I think we maybe just need to set some default values. I'll do a wee bit
more investigation.

Cheers.

 

On 18/10/2013 14:43, "helix84"  wrote:

>I can confirm I'm seeing the same thing in DSpace 4 and that DSpace 3
>had actual numbers there. I join you in your confusion.
>
>Can you observe any ill effects? I didn't notice anything wrong and I
>ran "update-discovery-index -f" on ~20k items yesterday.
>
>
>Regards,
>~~helix84
>
>Compulsory reading: DSpace Mailing List Etiquette
>https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr config parameter interpolation

2013-10-18 Thread helix84 
I can confirm I'm seeing the same thing in DSpace 4 and that DSpace 3
had actual numbers there. I join you in your confusion.

Can you observe any ill effects? I didn't notice anything wrong and I
ran "update-discovery-index -f" on ~20k items yesterday.


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

[Dspace-tech] Solr config parameter interpolation

2013-10-18 Thread TAYLOR Robin 
Hi all,

I'm playing around with the Dspace 4.0 code and it seems like my Solr Discovery 
index is not getting updated as I would expect. Looking at the Solr config file 
I see:-

 
   ${solr.autoCommit.maxTime:15000}
   false
 



 
   ${solr.autoSoftCommit.maxTime:-1}
 



Is this what you would expect to see, or should these parameters have been 
interpolated at some point?

Cheers, Robin.




-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette