Re: DIH for sharded database?
you can put table names in a different table and use like this
entity name=tab query=select table_name from sometable
rootEntity=false
entity query=select * from ${tab.table_name}
-
-
-
-
-
/entity
/entity
-
Grijesh
--
View this message in context:
http://lucene.472066.n3.nabble.com/DIH-for-sharded-database-tp2113767p2119370.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH for sharded database?
The easiest way, and the way that the database needs to use those shards,
probably, is to use a view with a queiry and I think it joins on the primary
key.
Dennis Gearon
Signature Warning
It is always a good idea to learn from your own mistakes. It is usually a
better
idea to learn from others’ mistakes, so you do not have to make them yourself.
from 'http://blogs.techrepublic.com.com/security/?p=4501tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
- Original Message
From: Andy angelf...@yahoo.com
To: solr-user@lucene.apache.org
Sent: Sat, December 18, 2010 6:20:54 PM
Subject: DIH for sharded database?
I have a table that is broken up into many virtual shards. So basically I have
N
identical tables:
Document1
Document2
.
.
Document36
Currently these tables all live in the same database, but in the future they
may
be moved to different servers to scale out if the needs arise.
Is there any way to configure a DIH for these tables so that it will
automatically loop through the 36 identical tables and pull data out for
indexing?
Something like (pseudo code):
for (i = 1; i = 36; i++) {
## retrieve data from the table Document{$i} index the data
}
What's the best way to handle a situation like this?
Thanks
Re: DIH for sharded database?
Some talk on giant databases in postgres:
http://wiki.postgresql.org/images/3/38/PGDay2009-EN-Datawarehousing_with_PostgreSQL.pdf
wikipedia
http://en.wikipedia.org/wiki/Partition_%28database%29
(says to use a UNION)
postgres description on how to do it:
http://www.postgresql.org/docs/current/interactive/ddl-partitioning.html
Dennis Gearon
Signature Warning
It is always a good idea to learn from your own mistakes. It is usually a
better
idea to learn from others’ mistakes, so you do not have to make them yourself.
from 'http://blogs.techrepublic.com.com/security/?p=4501tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
- Original Message
From: Andy angelf...@yahoo.com
To: solr-user@lucene.apache.org
Sent: Sat, December 18, 2010 6:20:54 PM
Subject: DIH for sharded database?
I have a table that is broken up into many virtual shards. So basically I have
N
identical tables:
Document1
Document2
.
.
Document36
Currently these tables all live in the same database, but in the future they
may
be moved to different servers to scale out if the needs arise.
Is there any way to configure a DIH for these tables so that it will
automatically loop through the 36 identical tables and pull data out for
indexing?
Something like (pseudo code):
for (i = 1; i = 36; i++) {
## retrieve data from the table Document{$i} index the data
}
What's the best way to handle a situation like this?
Thanks
Re: DIH for sharded database?
This is helpful. Thank you.
--- On Sun, 12/19/10, Dennis Gearon gear...@sbcglobal.net wrote:
From: Dennis Gearon gear...@sbcglobal.net
Subject: Re: DIH for sharded database?
To: solr-user@lucene.apache.org
Date: Sunday, December 19, 2010, 11:56 AM
Some talk on giant databases in
postgres:
http://wiki.postgresql.org/images/3/38/PGDay2009-EN-Datawarehousing_with_PostgreSQL.pdf
wikipedia
http://en.wikipedia.org/wiki/Partition_%28database%29
(says to use a UNION)
postgres description on how to do it:
http://www.postgresql.org/docs/current/interactive/ddl-partitioning.html
Dennis Gearon
Signature Warning
It is always a good idea to learn from your own mistakes.
It is usually a better
idea to learn from others’ mistakes, so you do not have
to make them yourself.
from 'http://blogs.techrepublic.com.com/security/?p=4501tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
- Original Message
From: Andy angelf...@yahoo.com
To: solr-user@lucene.apache.org
Sent: Sat, December 18, 2010 6:20:54 PM
Subject: DIH for sharded database?
I have a table that is broken up into many virtual shards.
So basically I have N
identical tables:
Document1
Document2
.
.
Document36
Currently these tables all live in the same database, but
in the future they may
be moved to different servers to scale out if the needs
arise.
Is there any way to configure a DIH for these tables so
that it will
automatically loop through the 36 identical tables and pull
data out for
indexing?
Something like (pseudo code):
for (i = 1; i = 36; i++) {
## retrieve data from the table
Document{$i} index the data
}
What's the best way to handle a situation like this?
Thanks
Re: DIH for sharded database?
You said: Currently these tables all live in the same database, but in the future they may be moved to different servers to scale out if the needs arise. That's why I concentrated on the JDBC url problem. But you can use a file as a list of tables. Read each line, and a sub-entity can substitute the line value into the SQL statement. On Sat, Dec 18, 2010 at 6:46 PM, Andy angelf...@yahoo.com wrote: --- On Sat, 12/18/10, Lance Norskog goks...@gmail.com wrote: You can have a file with 1,2,3 on separate lines. There is a line-by-line file reader that can pull these as separate drivers. Inside that entity the JDBC url has to be altered with the incoming numbers. I don't know if this will work. I'm not sure I understand. How will altering the JDBC url change the name of the table it is importing data from? Wouldn't I need to change the actual SQL query itself? select * from Document1 select * from Document2 ... select * from Document36 -- Lance Norskog goks...@gmail.com
Re: DIH for sharded database?
--- On Mon, 12/20/10, Lance Norskog goks...@gmail.com wrote: You said: Currently these tables all live in the same database, but in the future they may be moved to different servers to scale out if the needs arise. That's why I concentrated on the JDBC url problem. But you can use a file as a list of tables. Read each line, and a sub-entity can substitute the line value into the SQL statement. Can you give me an example of how to do this or pointing me to documentation that illustrates this? I think I sorta understand what you're saying conceptually but I need to be sure about the specifics. Thanks.
DIH for sharded database?
I have a table that is broken up into many virtual shards. So basically I have
N identical tables:
Document1
Document2
.
.
Document36
Currently these tables all live in the same database, but in the future they
may be moved to different servers to scale out if the needs arise.
Is there any way to configure a DIH for these tables so that it will
automatically loop through the 36 identical tables and pull data out for
indexing?
Something like (pseudo code):
for (i = 1; i = 36; i++) {
## retrieve data from the table Document{$i} index the data
}
What's the best way to handle a situation like this?
Thanks
Re: DIH for sharded database?
You can have a file with 1,2,3 on separate lines. There is a
line-by-line file reader that can pull these as separate drivers.
Inside that entity the JDBC url has to be altered with the incoming
numbers. I don't know if this will work.
It also may work for single-threaded DIH but not during multiple
threads. (Ignore this for Solr 1.4, you have no threads feature.)
On Sat, Dec 18, 2010 at 6:20 PM, Andy angelf...@yahoo.com wrote:
I have a table that is broken up into many virtual shards. So basically I
have N identical tables:
Document1
Document2
.
.
Document36
Currently these tables all live in the same database, but in the future they
may be moved to different servers to scale out if the needs arise.
Is there any way to configure a DIH for these tables so that it will
automatically loop through the 36 identical tables and pull data out for
indexing?
Something like (pseudo code):
for (i = 1; i = 36; i++) {
## retrieve data from the table Document{$i} index the data
}
What's the best way to handle a situation like this?
Thanks
--
Lance Norskog
goks...@gmail.com
Re: DIH for sharded database?
--- On Sat, 12/18/10, Lance Norskog goks...@gmail.com wrote: You can have a file with 1,2,3 on separate lines. There is a line-by-line file reader that can pull these as separate drivers. Inside that entity the JDBC url has to be altered with the incoming numbers. I don't know if this will work. I'm not sure I understand. How will altering the JDBC url change the name of the table it is importing data from? Wouldn't I need to change the actual SQL query itself? select * from Document1 select * from Document2 ... select * from Document36
