Re: Data Import Handler (DIH) - Installing and running

2020-12-23 Thread Erick Erickson
Have you done what the message says and looked at your Solr log? If so, what information is there? > On Dec 23, 2020, at 5:13 AM, DINSD | SPAutores > wrote: > > Hi, > > I'm trying to install the package "data-import-handler", since it was > discontinu

Data Import Handler (DIH) - Installing and running

2020-12-23 Thread DINSD | SPAutores
Hi, I'm trying to install the package "data-import-handler", since it was discontinued from core SolR distro. https://github.com/rohitbemax/dataimporthandler However, as soon as the first command is carried out solr -c -Denable.packages=true I get this screen in web interf

Re: data import handler deprecated?

2020-11-30 Thread Dmitri Maziuk
On 11/30/2020 7:50 AM, David Smiley wrote: Yes, absolutely to what Eric said. We goofed on news / release highlights on how to communicate what's happening in Solr. From a Solr insider point of view, we are "deprecating" because strictly speaking, the code isn't in our codebase any longer. Fro

Re: data import handler deprecated?

2020-11-30 Thread David Smiley
Yes, absolutely to what Eric said. We goofed on news / release highlights on how to communicate what's happening in Solr. From a Solr insider point of view, we are "deprecating" because strictly speaking, the code isn't in our codebase any longer. From a user point of view (the audience of news

Re: data import handler deprecated?

2020-11-30 Thread Eric Pugh
You don’t need to abandon DIH right now…. You can just use the Github hosted version…. The more people who use it, the better a community it will form around it!It’s a bit chicken and egg, since no one is actively discussing it, submitting PR’s etc, it may languish. If you use it, and

Re: data import handler deprecated?

2020-11-29 Thread Dmitri Maziuk
On 11/29/2020 10:32 AM, Erick Erickson wrote: And I absolutely agree with Walter that the DB is often where the bottleneck lies. You might be able to use multiple threads and/or processes to query the DB if that’s the case and you can find some kind of partition key. IME the difficult part has

Re: data import handler deprecated?

2020-11-29 Thread Erick Erickson
If you like Java instead of Python, here’s a skeletal program: https://lucidworks.com/post/indexing-with-solrj/ It’s simple and single-threaded, but could serve as a basis for something along the lines that Walter suggests. And I absolutely agree with Walter that the DB is often where the bottle

Re: data import handler deprecated?

2020-11-29 Thread Walter Underwood
I recommend building an outboard loader, like I did a dozen years ago for Solr 1.3 (before DIH) and did again recently. I’m glad to send you my Python program, though it reads from a JSONL file, not a database. Run a loop fetching records from a database. Put each record into a synchronized (threa

Re: data import handler deprecated?

2020-11-28 Thread matthew sporleder
I went through the same stages of grief that you are about to start but (luckily?) my core dataset grew some weird cousins and we ended up writing our own indexer to join them all together/do partial updates/other stuff beyond DIH. It's not difficult to upload docs but is definitely slower so far.

Re: data import handler deprecated?

2020-11-28 Thread Dmitri Maziuk
On 11/28/2020 5:48 PM, matthew sporleder wrote: ... The bottom of that github page isn't hopeful however :) Yeah, "works with MariaDB" is a particularly bad way of saying "BYO JDBC JAR" :) It's a more general queston though, what is the path forward for users who with data in two places?

Re: data import handler deprecated?

2020-11-28 Thread matthew sporleder
https://solr.cool/#utilities -> https://github.com/rohitbemax/dataimporthandler You can import it in the many new/novel ways to add things to a solr install and it should work like always (apparently). The bottom of that github page isn't hopeful however :) On Sat, Nov 28, 2020 at 5:21 PM Dmitri

data import handler deprecated?

2020-11-28 Thread Dmitri Maziuk
Hi all, trying to set up solr-8.7.0, contrib/dataimporthandler/README.txt says this module is deprecated as of 8.6 and scheduled for removal in 9.0. How do we pull data out of our relational database in 8.7+? TIA Dima

Re: Data Import Handler - Concurrent Entity Importing

2020-05-13 Thread ART GALLERY
check out the videos on this website TROO.TUBE don't be such a sheep/zombie/loser/NPC. Much love! https://troo.tube/videos/watch/aaa64864-52ee-4201-922f-41300032f219 On Tue, May 5, 2020 at 1:58 PM Mikhail Khludnev wrote: > > Hello, James. > > DataImportHandler has a lock preventing concurrent exe

Re: Data Import Handler - Concurrent Entity Importing

2020-05-05 Thread Mikhail Khludnev
Hello, James. DataImportHandler has a lock preventing concurrent execution. If you need to run several imports in parallel at the same core, you need to duplicate "/dataimport" handlers definition in solrconfig.xml. Thus, you can run them in parallel. Regarding schema, I prefer the latter but mile

Data Import Handler - Concurrent Entity Importing

2020-05-05 Thread James Greene
Hello, I'm new to the group here so please excuse me if I do not have the etiquette down yet. Is it possible to have multiple entities (customer configurable, up to 40 atm) in a DIH configuration to be imported at once? Right now I have multiple root entities in my configuration but they get inde

SOLR Data Import Handler : A command is still running...

2020-02-03 Thread Doss
We are doing hourly data import to our index, per day one or two requests are getting failed with the message "A command is still running...". 1. Does it mean, the data import not happened for the last hour? 2. If you look at the "Full Dump Started" time has an older data, in the below log all mos

Re: SQL data import handler

2019-09-09 Thread Friscia, Michael
scia, Michael [mailto:michael.fris...@yale.edu] > Sent: Monday, September 09, 2019 1:22 PM > To: solr-user@lucene.apache.org > Subject: SQL data import handler > > I setup SOLR on Ubuntu 18.04 and installed Java from apt-get with default-jre > which installed versi

RE: SQL data import handler

2019-09-09 Thread Vadim Ivanov
; From: Friscia, Michael [mailto:michael.fris...@yale.edu] > Sent: Monday, September 09, 2019 1:22 PM > To: solr-user@lucene.apache.org > Subject: SQL data import handler > > I setup SOLR on Ubuntu 18.04 and installed Java from apt-get with default-jre > which installed version 11. So

Re: SQL data import handler

2019-09-09 Thread Jörn Franke
. Best regards > Am 09.09.2019 um 12:21 schrieb Friscia, Michael : > > I setup SOLR on Ubuntu 18.04 and installed Java from apt-get with default-jre > which installed version 11. So after a day of trying to make my Microsoft SQL > Server data import handler work and failing, I bui

SQL data import handler

2019-09-09 Thread Friscia, Michael
I setup SOLR on Ubuntu 18.04 and installed Java from apt-get with default-jre which installed version 11. So after a day of trying to make my Microsoft SQL Server data import handler work and failing, I built a new VM and installed JRE 8 and then everything works perfectly. The root of the

SV: data-import-handler for solr-7.5.0

2018-10-02 Thread Martin Frank Hansen (MHQ)
: Re: data-import-handler for solr-7.5.0 Ok, so then you can switch to debug mode and keep trying to figure it out. Also try BinFileDataSource or URLDataSource, maybe it will have an easier way. Or using relative path (example: https://github.com/arafalov/solr-apachecon2018-presentation/blob

Re: data-import-handler for solr-7.5.0

2018-10-02 Thread Alexandre Rafalovitch
Data, IM & Analytics > > > > Lautrupparken 40-42, DK-2750 Ballerup > E-mail m...@kmd.dk Web www.kmd.dk > Mobil +4525571418 > > -Oprindelig meddelelse- > Fra: Alexandre Rafalovitch > Sendt: 2. oktober 2018 18:18 > Til: solr-user > Emne: Re: data-imp

SV: data-import-handler for solr-7.5.0

2018-10-02 Thread Martin Frank Hansen (MHQ)
1418 -Oprindelig meddelelse- Fra: Alexandre Rafalovitch Sendt: 2. oktober 2018 18:18 Til: solr-user Emne: Re: data-import-handler for solr-7.5.0 Admin UI for DIH will show you the config file read. So, if nothing is there, the path is most likely the issue You can also provide or update

SV: data-import-handler for solr-7.5.0

2018-10-02 Thread Martin Frank Hansen (MHQ)
;, "Total Documents Skipped":"0", "Full Dump Started":"2018-10-02 16:15:21", "":"Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.", "Committed":"2018-10-02 16:15:22", "Time taken&quo

Re: data-import-handler for solr-7.5.0

2018-10-02 Thread Alexandre Rafalovitch
Høydahl, search solution architect > Cominvent AS - www.cominvent.com > > > 2. okt. 2018 kl. 17:15 skrev Martin Frank Hansen (MHQ) : > > > > Hi, > > > > I am having some problems getting the data-import-handler in Solr to work. > > I have tried a lot of things b

Re: data-import-handler for solr-7.5.0

2018-10-02 Thread Jan Høydahl
> Hi, > > I am having some problems getting the data-import-handler in Solr to work. I > have tried a lot of things but I simply get no response from Solr, not even > an error. > > When calling the API: > http://localhost:8983/solr/nh/dataimport?command=full-import >

data-import-handler for solr-7.5.0

2018-10-02 Thread Martin Frank Hansen (MHQ)
Hi, I am having some problems getting the data-import-handler in Solr to work. I have tried a lot of things but I simply get no response from Solr, not even an error. When calling the API: http://localhost:8983/solr/nh/dataimport?command=full-import { "responseHeader":{

Re: Data Import Handler with Solr Source behind Load Balancer

2018-09-14 Thread Emir Arnautović
port Training - http://sematext.com/ > On 12 Sep 2018, at 05:53, Zimmermann, Thomas > wrote: > > We have a Solr v7 Instance sourcing data from a Data Import Handler with a > Solr data source running Solr v4. When it hits a single server in that > instance directly, all document

Data Import Handler with Solr Source behind Load Balancer

2018-09-11 Thread Zimmermann, Thomas
We have a Solr v7 Instance sourcing data from a Data Import Handler with a Solr data source running Solr v4. When it hits a single server in that instance directly, all documents are read and written correctly to the v7. When we hit the load balancer DNS entry, the resulting data import handler

Re: Child=true does not work for data import handler

2018-08-08 Thread omp...@rediffmail.com
But in my case i see output as below 0 0 *:* on xml 1533734431931 IT 1 1 1608130338704326656 Data 1 2 1608130338704326656 omkar 1 1608130338704326656 ITI 2 3 1608130338712715264 Entry 2

Re: Child=true does not work for data import handler

2018-08-08 Thread Mikhail Khludnev
This is how nested docs look like. These are document blocks with parent in the end. Block Join Queries work on these blocks. On Wed, Aug 8, 2018 at 12:47 PM omp...@rediffmail.com < omkar.pra...@gmail.com> wrote: > Thanks a lot Mikhail. But as per documentation below nested document > ingestion i

Re: Child=true does not work for data import handler

2018-08-08 Thread omp...@rediffmail.com
Thanks a lot Mikhail. But as per documentation below nested document ingestion is possible. Is this limitation of DIH? https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.html#UploadingDatawithIndexHandlers-NestedChildDocuments Also can block join query be used to get exp

Re: Child=true does not work for data import handler

2018-08-06 Thread Mikhail Khludnev
It never works like you expect. You need to search for parents and then hook up [child]. I see some improvements are coming, but now that is. On Mon, Aug 6, 2018 at 9:11 PM omp...@rediffmail.com wrote: > Thanks Mikhail verbose did help. _root_ field was missing in schema also in > make some chan

Re: Child=true does not work for data import handler

2018-08-06 Thread omp...@rediffmail.com
Thanks Mikhail verbose did help. _root_ field was missing in schema also in make some changes in child entity. Like i created id as alias to emp_id ( in child query) which is id column of parent table.

Re: Child=true does not work for data import handler

2018-08-06 Thread Mikhail Khludnev
DIH has debug&verbose modes. Have you tried to use them? On Mon, Aug 6, 2018 at 4:11 PM omp...@rediffmail.com wrote: > Thanks Mikhail, i tried changing conf but that did not help > > > driver="com.mysql.jdbc.Driver" > url="jdbc:mysql://localhost:3306/test" >

Re: Child=true does not work for data import handler

2018-08-06 Thread omp...@rediffmail.com
Thanks Mikhail, i tried changing conf but that did not help

Re: Child=true does not work for data import handler

2018-08-04 Thread Mikhail Khludnev
Hi, Omkar. Could it happen that child docs as well as parents are assigned same "id" field values implicitly and removed due to uniqueKey collision? On Sat, Aug 4, 2018 at 10:12 PM omkar.pra...@gmail.com < omkar.pra...@gmail.com> wrote: > I am using similar db-data config as below for indexing t

Child=true does not work for data import handler

2018-08-04 Thread omkar.pra...@gmail.com
I am using similar db-data config as below for indexing this parent-child data. solr version 6.6.2 SELECT id as emp_id, name FROM emp; +++ | emp_id | name | +++ | 1 | omkar | | 2 | ashwin | +++ 2 rows in set (0.00 sec) select * fro

How to use tika-OCR in data import handler?

2018-07-23 Thread Yasufumi Mizoguchi
Hi, I am trying to use tika-OCR(Tesseract) in data import handler and found that processing English documents was quite good. But I am struggling to process the other languages such as Japanese, Chinese, etc... So, I want to know how to switch Tesseract-OCR's processing language via data i

Re: How to know the name(url) of documents that data import handler skipped

2018-07-08 Thread Yasufumi Mizoguchi
; > > I am trying to indexing files into Solr 7.2 using data import handler > with > > onError=skip option. > > But, I am struggling with determining the skipped documents as logs do > not > > tell which file was bad. > > So, how can I know those files? > > > > Thanks, > > Yasufumi >

Re: How to know the name(url) of documents that data import handler skipped

2018-07-08 Thread Rahul Singh
Have you tried changing the log level https://lucene.apache.org/solr/guide/7_2/configuring-logging.html -- Rahul Singh rahul.si...@anant.us Anant Corporation On Jul 8, 2018, 8:54 PM -0500, Yasufumi Mizoguchi , wrote: > Hi, > > I am trying to indexing files into Solr 7.2 using da

How to know the name(url) of documents that data import handler skipped

2018-07-08 Thread Yasufumi Mizoguchi
Hi, I am trying to indexing files into Solr 7.2 using data import handler with onError=skip option. But, I am struggling with determining the skipped documents as logs do not tell which file was bad. So, how can I know those files? Thanks, Yasufumi

RE: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-26 Thread msaunier
mysql jdbc connector version I need ? -Message d'origine- De : msaunier [mailto:msaun...@citya.com] Envoyé : jeudi 26 avril 2018 13:13 À : solr-user@lucene.apache.org Objet : RE: SolrCloud DIH (Data Import Handler) MySQL 404 Hello, Where I add that? In the Solr start comman

RE: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-26 Thread msaunier
: Re: SolrCloud DIH (Data Import Handler) MySQL 404 Can you share more log lines around this odd NPE? It might be necessary to restart jvm with -verbose:class and look through its' output to find why it can't load this class. On Wed, Apr 25, 2018 at 11:42 AM, msaunier wrote: > Hello

Re: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-25 Thread Mikhail Khludnev
o:elyog...@elyograg.org] > Envoyé : mardi 24 avril 2018 17:39 > À : solr-user@lucene.apache.org > Objet : Re: SolrCloud DIH (Data Import Handler) MySQL 404 > > On 4/24/2018 2:03 AM, msaunier wrote: > > If I access to the interface, I have a null pointer exception: > > &g

RE: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-25 Thread msaunier
: Re: SolrCloud DIH (Data Import Handler) MySQL 404 On 4/24/2018 2:03 AM, msaunier wrote: > If I access to the interface, I have a null pointer exception: > > null:java.lang.NullPointerException > at > org.apache.solr.handler.RequestHandlerBase.getVersion(RequestHandlerBa > se

Re: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-24 Thread Shawn Heisey
On 4/24/2018 2:03 AM, msaunier wrote: If I access to the interface, I have a null pointer exception: null:java.lang.NullPointerException at org.apache.solr.handler.RequestHandlerBase.getVersion(RequestHandlerBase.java:233) The line of code where this exception occurred uses fundamenta

RE: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-24 Thread msaunier
di 24 avril 2018 10:04 À : solr-user@lucene.apache.org Objet : RE: SolrCloud DIH (Data Import Handler) MySQL 404 If I access to the interface, I have a null pointer exception: null:java.lang.NullPointerException at org.apache.solr.handler.RequestHandlerBase.getVersion(RequestHandlerBase

RE: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-24 Thread msaunier
a.com] Envoyé : mardi 24 avril 2018 09:25 À : solr-user@lucene.apache.org Objet : RE: SolrCloud DIH (Data Import Handler) MySQL 404 Hello Shawn, Thanks for your answers. # So, indexation_events.xml fi

RE: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-24 Thread msaunier
or-java":{ "name":"mysql-connector-java", "version":1}, "data-import-handler":{ "name":"data-import-handler", "version":1}}, "requestHandler":{"/test_dih":{ "n

Re: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-23 Thread Mikhail Khludnev
st.java:95) > at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run( > SelectChannelEndPoint.java: > 93) > at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume. > executeProduceC > onsume(ExecuteProduceConsume.java:303) > at >

Re: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-23 Thread Shawn Heisey
On 4/23/2018 8:30 AM, msaunier wrote: I have add debug: curl "http://srv-formation-solr:8983/solr/arguments_test/test_dih?command=full-im port&commit=true&debug=true" 500588true1DIH/indexation_events.xml This is looking like a really nasty error that I cannot understand, possibly caused by a

Re: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-23 Thread Shawn Heisey
On 4/23/2018 6:12 AM, msaunier wrote: I have a problem with DIH in SolrCloud. I don't understand why, so I need your help. Solr 6.6 in Cloud. ## COMMAND: curl http://srv-formation-solr:8983/solr/test_dih?command=full-import RESULT: Error 404 Not F

RE: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-23 Thread msaunier
] Envoyé : lundi 23 avril 2018 14:47 À : solr-user@lucene.apache.org Objet : RE: SolrCloud DIH (Data Import Handler) MySQL 404 I have correct url to : curl http://srv-formation-solr:8983/solr/arguments_test/test_dih?command=full-imp ort And change overlay config "/configs/arguments_test/DIH/ind

RE: SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-23 Thread msaunier
Envoyé : lundi 23 avril 2018 14:12 À : solr-user@lucene.apache.org Objet : SolrCloud DIH (Data Import Handler) MySQL 404 Hello, I have a problem with DIH in SolrCloud. I don't understand why, so I need your help. Solr 6.6 in Cloud. ## COMMAND: curl

SolrCloud DIH (Data Import Handler) MySQL 404

2018-04-23 Thread msaunier
e add data-import-handler and mysql-connector-java runtimeLib on the configoverlay.json file with the API 4. I have create the DIH folder on the cloud with zkcli.sh script 5. I have push with zkcli the DIH .xml configuration file CONFIGOVERLAY CONTENT : { "runtimeLib"

Re: Solr Search Problem with Multiple Data-Import Handler

2017-06-22 Thread Josh Lincoln
I suspect Erik's right that clean=true is the problem. That's the default in the DIH interface. I find that when using DIH, it's best to set preImportDeleteQuery for every entity. This safely scopes the clean variable to just that entity. It doesn't look like the docs have examples of using preIm

Re: Solr Search Problem with Multiple Data-Import Handler

2017-06-21 Thread Erick Erickson
First place I'd look is whether the jobs have clean=true set. If so the first thing DIH does is delete all documents. Best, Erick On Wed, Jun 21, 2017 at 3:52 PM, Pandey Brahmdev wrote: > Hi, > I have setup Apache Solr 6.6.0 on Windows 10, 64-bit. > > I have created a simple core & configured D

Solr Search Problem with Multiple Data-Import Handler

2017-06-21 Thread Pandey Brahmdev
Hi, I have setup Apache Solr 6.6.0 on Windows 10, 64-bit. I have created a simple core & configured DataImport Handlers. I have configured 2 dataImport handlers in the Solr-config.xml file. First for to connect to DB & have data from DB Tables. And Second for to have data from all pdf files using

Data import handler and no status in web-ui

2017-06-06 Thread Thomas Porschberg
Hi, I use DIH in solr-cloud mode (implicit route) in solr6.5.1. When I start the import it works fine and I see the progress in the logfile. However, when I click the "Refresh Status" button in the web-ui while the import is running I only see "No information available (idle)". So I have to look

RE: Using the Data Import Handler with SQLite

2017-05-23 Thread Dheeraj Kumar Karnati
Hi Zac, I think you have added entity closing tag 2 times. that might be causing an issue. It been a long time . not sure whether you are still working on it or not. -- View this message in context: http://lucene.472066.n3.nabble.com/Using-the-Data-Import-Handler-with-SQLite

Re: Data Import Handler on 6.4.1

2017-03-15 Thread Walter Underwood
.1448 / > daphne@cevalogistics.com > > > -Original Message- > From: Michael Tobias [mailto:mtob...@btinternet.com] > Sent: Wednesday, March 15, 2017 2:36 PM > To: solr-user@lucene.apache.org > Subject: Data Import Handler on 6.4.1 > > I am sure I am missin

RE: Data Import Handler on 6.4.1

2017-03-15 Thread Liu, Daphne
904.928.1448 / daphne@cevalogistics.com -Original Message- From: Michael Tobias [mailto:mtob...@btinternet.com] Sent: Wednesday, March 15, 2017 2:36 PM To: solr-user@lucene.apache.org Subject: Data Import Handler on 6.4.1 I am sure I am missing something simple but I am running

Data Import Handler on 6.4.1

2017-03-15 Thread Michael Tobias
I am sure I am missing something simple but I am running Solr 4.8.1 and trialling 6.4.1 on another computer. I have had to manually modify the automatic 6.4.1 scheme config as we use a set of specialised field types. They work fine. I am now trying to populate my core with data and having

Re: Data Import Handler, also "Real Time" index updates

2017-03-05 Thread Damien Kamerman
You could configure the dataimporthandler to not delete at the start (either do a delta or set the preimportdeltequery), and set a postimportdeletequery if required. On Saturday, 4 March 2017, Alexandre Rafalovitch wrote: > Commit is index global. So if you have overlapping timelines and commit

Re: Data Import Handler, also "Real Time" index updates

2017-03-03 Thread Alexandre Rafalovitch
Commit is index global. So if you have overlapping timelines and commit is issued, it will affect all changes done to that point. So, the aliases may be better for you. You could potentially also reload a cure with changes solrconfig.XML settings, but that's heavy on caches. Regards, Alex On

Re: Data Import Handler, also "Real Time" index updates

2017-03-03 Thread Sales
> > You have indicated that you have a way to avoid doing updates during the > full import. Because of this, you do have another option that is likely > much easier for you to implement: Set the "commitWithin" parameter on > each update request. This works almost identically to autoSoftCommit,

Re: Data Import Handler, also "Real Time" index updates

2017-03-03 Thread Shawn Heisey
On 3/3/2017 10:17 AM, Sales wrote: > I am not sure how best to handle this. We use the data import handle to > re-sync all our data on a daily basis, takes 1-2 hours depending on system > load. It is set up to commit at the end, so, the old index remains until it’s > done, and, we lose no access

Re: Data Import Handler, also "Real Time" index updates

2017-03-03 Thread Sales
> On Mar 3, 2017, at 11:30 AM, Erick Erickson wrote: > > One way to handle this (presuming SolrCloud) is collection aliasing. > You create two collections, c1 and c2. You then have two aliases. when > you start "index" is aliased to c1 and "search" is aliased to c2. Now > do your full import to

Re: Data Import Handler, also "Real Time" index updates

2017-03-03 Thread Erick Erickson
One way to handle this (presuming SolrCloud) is collection aliasing. You create two collections, c1 and c2. You then have two aliases. when you start "index" is aliased to c1 and "search" is aliased to c2. Now do your full import to "index" (and, BTW, you'd be well advised to do at least a hard co

Re: Data Import Handler, also "Real Time" index updates

2017-03-03 Thread Sales
> > On Mar 3, 2017, at 11:22 AM, Alexandre Rafalovitch wrote: > > On 3 March 2017 at 12:17, Sales > wrote: >> When we enabled those, during the index, the data disappeared since it kept >> soft committing during the import process, > > This part does not quite make sense. Could you expand on

Re: Data Import Handler, also "Real Time" index updates

2017-03-03 Thread Alexandre Rafalovitch
On 3 March 2017 at 12:17, Sales wrote: > When we enabled those, during the index, the data disappeared since it kept > soft committing during the import process, This part does not quite make sense. Could you expand on this "data disappeared" part to understand what the issue is. The main issue

Data Import Handler, also "Real Time" index updates

2017-03-03 Thread Sales
I am not sure how best to handle this. We use the data import handle to re-sync all our data on a daily basis, takes 1-2 hours depending on system load. It is set up to commit at the end, so, the old index remains until it’s done, and, we lose no access while the import is happening. But, we no

Re: Solr Data Import Handler

2017-02-12 Thread Alexandre Rafalovitch
They don't have to wait because they are not synchronized on anything AFAIK. I do believe they need to be registered as different request handler names in solrconfig.xml But you need to be careful if they both/all wipe out data before indexing new one. Those calls could overlap. Regards, Alex.

Re: Solr Data Import Handler

2017-02-12 Thread Dave
That sounds pretty much like a hack. So if two imports happen at the same time they have to wait for each other? > On Feb 12, 2017, at 4:01 PM, Shawn Heisey wrote: > >> On 2/12/2017 10:30 AM, Minh wrote: >> Hi everyone, >> How can i run multithreads of DIH in a cluster for a collection? > > Th

Re: Solr Data Import Handler

2017-02-12 Thread Shawn Heisey
On 2/12/2017 10:30 AM, Minh wrote: > Hi everyone, > How can i run multithreads of DIH in a cluster for a collection? The DIH handler is single-threaded. It used to have a config option for multiple threads, but it was removed since it didn't actually work. If you create multiple DIH handlers and

Solr Data Import Handler

2017-02-12 Thread Minh
Hi everyone, How can i run multithreads of DIH in a cluster for a collection?

Re: problem with data import handler delta import due to use of multiple datasource

2017-01-17 Thread amylindan
Did you solve the problem? I'm stuck with exactly the same problem now. Let me know if you already had a solution,please. -- View this message in context: http://lucene.472066.n3.nabble.com/problem-with-data-import-handler-delta-import-due-to-use-of-multiple-datasource-tp4093698p4314273

Re: Data Import Handler - maximum?

2016-12-12 Thread Shawn Heisey
On 12/11/2016 8:00 PM, Brian Narsi wrote: > We are using Solr 5.1.0 and DIH to build index. > > We are using DIH with clean=true and commit=true and optimize=true. > Currently retrieving about 10.5 million records in about an hour. > > I will like to find from other member's experiences as to how l

Re: Data Import Handler - maximum?

2016-12-12 Thread Bernd Fehling
Am 12.12.2016 um 04:00 schrieb Brian Narsi: > We are using Solr 5.1.0 and DIH to build index. > > We are using DIH with clean=true and commit=true and optimize=true. > Currently retrieving about 10.5 million records in about an hour. > > I will like to find from other member's experiences as to

Data Import Handler - maximum?

2016-12-11 Thread Brian Narsi
We are using Solr 5.1.0 and DIH to build index. We are using DIH with clean=true and commit=true and optimize=true. Currently retrieving about 10.5 million records in about an hour. I will like to find from other member's experiences as to how long can DIH run with no issues? What is the maximum

Re: Scheduling Data Import Handler (DIH) Delta Imports

2016-11-28 Thread Walter Underwood
First, try to do it with something like Apache Camel. That moves the whole database import process outside of Solr where it can be more easily controlled. http://camel.apache.org/ wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog)

Re: Scheduling Data Import Handler (DIH) Delta Imports

2016-11-28 Thread Jamie Jackson
One last bump before I get crackin'... On Mon, Nov 21, 2016 at 11:54 AM, Jamie Jackson wrote: > Hi Folks, > > I have DIH cores that are being indexed by my Lucee application. That > works, but I'd like to make some improvements: > > >- Make a standalone scheduler that's not part of a larger

Scheduling Data Import Handler (DIH) Delta Imports

2016-11-21 Thread Jamie Jackson
Hi Folks, I have DIH cores that are being indexed by my Lucee application. That works, but I'd like to make some improvements: - Make a standalone scheduler that's not part of a larger application. (FYI, I want to Dockerize the import-triggering service.) - Prevent import requests from

Re: Problem with Password Decryption in Data Import Handler

2016-11-03 Thread William Bell
col.java:763) >> at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils. >> java:469) >> at org.mariadb.jdbc.Driver.connect(Driver.java:104) >> ... 16 more >> >> >> >> On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson >> wrote: >> &

Re: Problem with Password Decryption in Data Import Handler

2016-11-03 Thread William Bell
gt;> wrote: > >> > I haven't tried this but is it possible there is a new line at the end > >> in > >> > the file? > >> > > >> > If you did something like echo "" > file.txt then there would be a new > >> > line

Re: Problem with Password Decryption in Data Import Handler

2016-11-03 Thread Jamie Jackson
ocol. > > connectWithoutProxy(AbstractConnectProtocol.java:763) > > at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469) > > at org.mariadb.jdbc.Driver.connect(Driver.java:104) > > ... 16 more > > > > > > > > On Thu, Oct 6, 2016 at 2:

Re: Problem with Password Decryption in Data Import Handler

2016-11-02 Thread Fuad Efendi
> >> Alex. > >> > >> Newsletter and resources for Solr beginners and intermediates: > >> http://www.solr-start.com/ > >> > >> > >> On 6 October 2016 at 23:26, Kevin Risden > >> wrote: > >> >

Re: Problem with Password Decryption in Data Import Handler

2016-11-02 Thread Jamie Jackson
yption key (file content)? Because the code I am > >> looking at seems to expect it to be at most 100 characters. > >> > >> Regards, > >> Alex. > >> > >> Newsletter and resources for Solr beginners and intermediates: > >> http://www.solr-start.com/

Re: Problem with Password Decryption in Data Import Handler

2016-11-02 Thread Fuad Efendi
haracters are in the file. >> > >> > Kevin Risden >> > >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson >> wrote: >> > >> >> Hi Folks, >> >> >> >> (Using Solr 5.5.3.) >> >> >> >> As far

Re: Problem with Password Decryption in Data Import Handler

2016-11-02 Thread Fuad Efendi
> If you did something like echo "" > file.txt then there would be a new >> > line. Use echo -n "" > file.txt >> > >> > Also you should be able to check how many characters are in the file. >> > >> > Kevin Risden >&

Re: Problem with Password Decryption in Data Import Handler

2016-11-02 Thread Jamie Jackson
are in the file. >> > >> > Kevin Risden >> > >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson >> wrote: >> > >> >> Hi Folks, >> >> >> >> (Using Solr 5.5.3.) >> >> >> >> As far as I know, the o

Re: Problem with Password Decryption in Data Import Handler

2016-10-06 Thread Jamie Jackson
son > wrote: > > > >> Hi Folks, > >> > >> (Using Solr 5.5.3.) > >> > >> As far as I know, the only place where encrypted password use is > documented > >> is in > >> https://cwiki.apache.org/confluence/display/solr/ > >

Re: Problem with Password Decryption in Data Import Handler

2016-10-06 Thread Jamie Jackson
.) > > > > As far as I know, the only place where encrypted password use is > documented > > is in > > https://cwiki.apache.org/confluence/display/solr/ > > Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler, > > under the "Configuring the DIH C

Re: Problem with Password Decryption in Data Import Handler

2016-10-06 Thread Alexandre Rafalovitch
rs are in the file. > > Kevin Risden > > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson wrote: > >> Hi Folks, >> >> (Using Solr 5.5.3.) >> >> As far as I know, the only place where encrypted password use is documented >> is in >> https://c

Re: Problem with Password Decryption in Data Import Handler

2016-10-06 Thread Kevin Risden
isden On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson wrote: > Hi Folks, > > (Using Solr 5.5.3.) > > As far as I know, the only place where encrypted password use is documented > is in > https://cwiki.apache.org/confluence/display/solr/ > Uploading+Structured+Data+Store+Data

Problem with Password Decryption in Data Import Handler

2016-10-05 Thread Jamie Jackson
Hi Folks, (Using Solr 5.5.3.) As far as I know, the only place where encrypted password use is documented is in https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler, under the "Configuring the DIH Configuration File", in a

RE: Data import handler in techproducts example

2016-07-07 Thread Brooks Chuck (FCA)
Hello Jonas, Did you figure this out? Dr. Chuck Brooks 248-838-5070 -Original Message- From: Jonas Vasiliauskas [mailto:jonas.vasiliaus...@yahoo.com.INVALID] Sent: Saturday, July 02, 2016 11:37 AM To: solr-user@lucene.apache.org Subject: Data import handler in techproducts example

Re: Data import handler in techproducts example

2016-07-02 Thread Ahmet Arslan
, 2016 9:56 PM, Jonas Vasiliauskas wrote: Hey, I'm quite new to solr and java environments. I have a goal for myself to import some data from mysql database in techproducts (core) example. I have setup data import handler (DIH) for techproducts based on instructions here

Data import handler in techproducts example

2016-07-02 Thread Jonas Vasiliauskas
Hey, I'm quite new to solr and java environments. I have a goal for myself to import some data from mysql database in techproducts (core) example. I have setup data import handler (DIH) for techproducts based on instructions here https://wiki.apache.org/solr/DIHQuickStart , but looks

  1   2   3   4   >