Any other customizations you are making to solrconfig?
On Mar 1, 2012, at 1:48 PM, Matthew Parker wrote:
> Added it back in. I still get the same result.
>
> On Wed, Feb 29, 2012 at 10:09 PM, Mark Miller <markrmil...@gmail.com> wrote:
> Do you have a _version_ field in your schema? I actually just came back to
> this thread with that thought and then saw your error - so that remains my
> guess.
>
> I'm going to improve the doc on the wiki around what needs to be defined
> for SolrCloud - so far we have things in the example defaults, but its not
> clear enough to users what needs to be there if they are using an old
> schema or modifying the example.
>
> <field name="_version_" type="long" indexed="true" stored="true"/>
>
> If you do have a _version_ field, there is something to track down here for
> sure.
>
> On Wed, Feb 29, 2012 at 1:15 PM, Matthew Parker <
> mpar...@apogeeintegration.com> wrote:
>
> > Mark/Sami
> >
> > I ran the system with 3 zookeeper nodes, 2 solr cloud nodes, and left
> > numShards set to its default value (i.e. 1)
> >
> > I looks like it finally sync'd with the other one after quite a while, but
> > it's throwing lots of errors like the following:
> >
> > org.apache.solr.common.SolrException: missing _version_ on update from
> > leader at
> > org.apache.solr.update.processor.DistributtedUpdateProcessor.versionDelete(
> > DistributedUpdateProcessor.java: 712)
> > ....
> > ....
> > ....
> >
> > Is it normal to sync long after the documents were sent for indexing?
> >
> > I'll have to check and see whether the 4 solr node instance with 2 shards
> > works after waiting for the system to sync.
> >
> > Regards,
> >
> > Matt
> >
> > On Wed, Feb 29, 2012 at 12:03 PM, Matthew Parker <
> > mpar...@apogeeintegration.com> wrote:
> >
> > > I also took out my requestHandler and used the standard /update/extract
> > > handler. Same result.
> > >
> > > On Wed, Feb 29, 2012 at 11:47 AM, Matthew Parker <
> > > mpar...@apogeeintegration.com> wrote:
> > >
> > >> I tried running SOLR Cloud with the default number of shards (i.e. 1),
> > >> and I get the same results.
> > >>
> > >> On Wed, Feb 29, 2012 at 10:46 AM, Matthew Parker <
> > >> mpar...@apogeeintegration.com> wrote:
> > >>
> > >>> Mark,
> > >>>
> > >>> Nothing appears to be wrong in the logs. I wiped the indexes and
> > >>> imported 37 files from SharePoint using Manifold. All 37 make it in,
> > but
> > >>> SOLR still has issues with the results being inconsistent.
> > >>>
> > >>> Let me run my setup by you, and see whether that is the issue?
> > >>>
> > >>> On one machine, I have three zookeeper instances, four solr instances,
> > >>> and a data directory for solr and zookeeper config data.
> > >>>
> > >>> Step 1. I modified each zoo.xml configuration file to have:
> > >>>
> > >>> Zookeeper 1 - Create /zookeeper1/conf/zoo.cfg
> > >>> ================
> > >>> tickTime=2000
> > >>> initLimit=10
> > >>> syncLimit=5
> > >>> dataDir=[DATA_DIRECTORY]/zk1_data
> > >>> clientPort=2181
> > >>> server.1=localhost:2888:3888
> > >>> server.2=localhost:2889:3889
> > >>> server.3=localhost:2890:3890
> > >>>
> > >>> Zookeeper 1 - Create /[DATA_DIRECTORY]/zk1_data/myid with the following
> > >>> contents:
> > >>> ==============================================================
> > >>> 1
> > >>>
> > >>> Zookeep 2 - Create /zookeeper2/conf/zoo.cfg
> > >>> ==============
> > >>> tickTime=2000
> > >>> initLimit=10
> > >>> syncLimit=5
> > >>> dataDir=[DATA_DIRECTORY]/zk2_data
> > >>> clientPort=2182
> > >>> server.1=localhost:2888:3888
> > >>> server.2=localhost:2889:3889
> > >>> server.3=localhost:2890:3890
> > >>>
> > >>> Zookeeper 2 - Create /[DATA_DIRECTORY]/zk2_data/myid with the following
> > >>> contents:
> > >>> ==============================================================
> > >>> 2
> > >>>
> > >>> Zookeeper 3 - Create /zookeeper3/conf/zoo.cfg
> > >>> ================
> > >>> tickTime=2000
> > >>> initLimit=10
> > >>> syncLimit=5
> > >>> dataDir=[DATA_DIRECTORY]/zk3_data
> > >>> clientPort=2183
> > >>> server.1=localhost:2888:3888
> > >>> server.2=localhost:2889:3889
> > >>> server.3=localhost:2890:3890
> > >>>
> > >>> Zookeeper 3 - Create /[DATA_DIRECTORY]/zk3_data/myid with the following
> > >>> contents:
> > >>> ====================================================
> > >>> 3
> > >>>
> > >>> Step 2 - SOLR Build
> > >>> ===============
> > >>>
> > >>> I pulled the latest SOLR trunk down. I built it with the following
> > >>> commands:
> > >>>
> > >>> ant example dist
> > >>>
> > >>> I modified the solr.war files and added the solr cell and extraction
> > >>> libraries to WEB-INF/lib. I couldn't get the extraction to work
> > >>> any other way. Will zookeper pickup jar files stored with the rest of
> > >>> the configuration files in Zookeeper?
> > >>>
> > >>> I copied the contents of the example directory to each of my SOLR
> > >>> directories.
> > >>>
> > >>> Step 3 - Starting Zookeeper instances
> > >>> ===========================
> > >>>
> > >>> I ran the following commands to start the zookeeper instances:
> > >>>
> > >>> start .\zookeeper1\bin\zkServer.cmd
> > >>> start .\zookeeper2\bin\zkServer.cmd
> > >>> start .\zookeeper3\bin\zkServer.cmd
> > >>>
> > >>> Step 4 - Start Main SOLR instance
> > >>> ==========================
> > >>> I ran the following command to start the main SOLR instance
> > >>>
> > >>> java -Djetty.port=8081 -Dhostport=8081
> > >>> -Dbootstrap_configdir=[DATA_DIRECTORY]/solr/conf -Dnumshards=2
> > >>> -Dzkhost=localhost:2181,localhost:2182,localhost:2183 -jar start.jar
> > >>>
> > >>> Starts up fine.
> > >>>
> > >>> Step 5 - Start the Remaining 3 SOLR Instances
> > >>> ==================================
> > >>> I ran the following commands to start the other 3 instances from their
> > >>> home directories:
> > >>>
> > >>> java -Djetty.port=8082 -Dhostport=8082
> > >>> -Dzkhost=localhost:2181,localhost:2182,localhost:2183 -jar start.jar
> > >>>
> > >>> java -Djetty.port=8083 -Dhostport=8083
> > >>> -Dzkhost=localhost:2181,localhost:2182,localhost:2183 -jar start.jar
> > >>>
> > >>> java -Djetty.port=8084 -Dhostport=8084
> > >>> -Dzkhost=localhost:2181,localhost:2182,localhost:2183 -jar start.jar
> > >>>
> > >>> All startup without issue.
> > >>>
> > >>> Step 6 - Modified solrconfig.xml to have a custom request handler
> > >>> ===============================================
> > >>>
> > >>> <requestHandler name="/update/sharepoint" startup="lazy"
> > >>> class="solr.extraction.ExtractingRequestHandler">
> > >>> <lst name="defaults">
> > >>> <str name="update.chain">sharepoint-pipeline</str>
> > >>> <str name="fmap.content">text</str>
> > >>> <str name="lowernames">true</str>
> > >>> <str name="uprefix">ignored</str>
> > >>> <str name="caputreAttr">true</str>
> > >>> <str name="fmap.a">links</str>
> > >>> <str name="fmap.div">ignored</str>
> > >>> </lst>
> > >>> </requestHandler>
> > >>>
> > >>> <updateRequestProcessorChain name="sharepoint-pipeline">
> > >>> <processor class="solr.processor.SignatureUpdateProcessorFactory">
> > >>> <bool name="enabled">true</bool>
> > >>> <str name="signatureField">id</str>
> > >>> <bool name="owerrightDupes">true</bool>
> > >>> <str name="fields">url</str>
> > >>> <str name="signatureClass">solr.processor.Lookup3Signature</str>
> > >>> </processor>
> > >>> <processor class="solr.LogUpdateProcessorFactory"/>
> > >>> <processor class="solr.RunUpdateProcessorFactory"/>
> > >>> </updateRequestProcessorChain>
> > >>>
> > >>>
> > >>> Hopefully this will shed some light on why my configuration is having
> > >>> issues.
> > >>>
> > >>> Thanks for your help.
> > >>>
> > >>> Matt
> > >>>
> > >>>
> > >>>
> > >>> On Tue, Feb 28, 2012 at 8:29 PM, Mark Miller <markrmil...@gmail.com
> > >wrote:
> > >>>
> > >>>> Hmm...this is very strange - there is nothing interesting in any of
> > the
> > >>>> logs?
> > >>>>
> > >>>> In clusterstate.json, all of the shards have an active state?
> > >>>>
> > >>>>
> > >>>> There are quite a few of us doing exactly this setup recently, so
> > there
> > >>>> must be something we are missing here...
> > >>>>
> > >>>> Any info you can offer might help.
> > >>>>
> > >>>> - Mark
> > >>>>
> > >>>> On Feb 28, 2012, at 1:00 PM, Matthew Parker wrote:
> > >>>>
> > >>>> > Mark,
> > >>>> >
> > >>>> > I got the codebase from the 2/26/2012, and I got the same
> > inconsistent
> > >>>> > results.
> > >>>> >
> > >>>> > I have solr running on four ports 8081-8084
> > >>>> >
> > >>>> > 8081 and 8082 are the leaders for shard 1, and shard 2, respectively
> > >>>> >
> > >>>> > 8083 - is assigned to shard 1
> > >>>> > 8084 - is assigned to shard 2
> > >>>> >
> > >>>> > queries come in and sometime it seems the windows from 8081 and 8083
> > >>>> move
> > >>>> > responding to the query but there are no results.
> > >>>> >
> > >>>> > if the queries run on 8081/8082 or 8081/8084 then results come back
> > >>>> ok.
> > >>>> >
> > >>>> > The query is nothing more than: q=*:*
> > >>>> >
> > >>>> > Regards,
> > >>>> >
> > >>>> > Matt
> > >>>> >
> > >>>> >
> > >>>> > On Mon, Feb 27, 2012 at 9:26 PM, Matthew Parker <
> > >>>> > mpar...@apogeeintegration.com> wrote:
> > >>>> >
> > >>>> >> I'll have to check on the commit situation. We have been pushing
> > >>>> data from
> > >>>> >> SharePoint the last week or so. Would that somehow block the
> > >>>> documents
> > >>>> >> moving between the solr instances?
> > >>>> >>
> > >>>> >> I'll try another version tomorrow. Thanks for the suggestions.
> > >>>> >>
> > >>>> >> On Mon, Feb 27, 2012 at 5:34 PM, Mark Miller <
> > markrmil...@gmail.com
> > >>>> >wrote:
> > >>>> >>
> > >>>> >>> Hmmm...all of that looks pretty normal...
> > >>>> >>>
> > >>>> >>> Did a commit somehow fail on the other machine? When you view the
> > >>>> stats
> > >>>> >>> for the update handler, are there a lot of pending adds for on of
> > >>>> the
> > >>>> >>> nodes? Do the commit counts match across nodes?
> > >>>> >>>
> > >>>> >>> You can also query an individual node with distrib=false to check
> > >>>> that.
> > >>>> >>>
> > >>>> >>> If you build is a month old, I'd honestly recommend you try
> > >>>> upgrading as
> > >>>> >>> well.
> > >>>> >>>
> > >>>> >>> - Mark
> > >>>> >>>
> > >>>> >>> On Feb 27, 2012, at 3:34 PM, Matthew Parker wrote:
> > >>>> >>>
> > >>>> >>>> Here is most of the cluster state:
> > >>>> >>>>
> > >>>> >>>> Connected to Zookeeper
> > >>>> >>>> localhost:2181, localhost: 2182, localhost:2183
> > >>>> >>>>
> > >>>> >>>> /(v=0 children=7) ""
> > >>>> >>>> /CONFIGS(v=0, children=1)
> > >>>> >>>> /CONFIGURATION(v=0 children=25)
> > >>>> >>>> <<<<< all the configuration files, velocity info,
> > xslt,
> > >>>> etc.
> > >>>> >>>>>>>>
> > >>>> >>>> /NODE_STATES(v=0 children=4)
> > >>>> >>>> MACHINE1:8083_SOLR (v=121)"[{"shard_id":"shard1",
> > >>>> >>>>
> > >>>> "state":"active","core":"","collection":"collection1","node_name:"..."
> > >>>> >>>> MACHINE1:8082_SOLR (v=101)"[{"shard_id":"shard2",
> > >>>> >>>>
> > >>>> "state":"active","core":"","collection":"collection1","node_name:"..."
> > >>>> >>>> MACHINE1:8081_SOLR (v=92)"[{"shard_id":"shard1",
> > >>>> >>>>
> > >>>> "state":"active","core":"","collection":"collection1","node_name:"..."
> > >>>> >>>> MACHINE1:8084_SOLR (v=73)"[{"shard_id":"shard2",
> > >>>> >>>>
> > >>>> "state":"active","core":"","collection":"collection1","node_name:"..."
> > >>>> >>>> /ZOOKEEPER (v-0 children=1)
> > >>>> >>>> QUOTA(v=0)
> > >>>> >>>>
> > >>>> >>>>
> > >>>> >>>
> > >>>>
> > /CLUSTERSTATE.JSON(V=272)"{"collection1":{"shard1":{MACHINE1:8081_solr_":{shard_id":"shard1","leader":"true","..."
> > >>>> >>>> /LIVE_NODES (v=0 children=4)
> > >>>> >>>> MACHINE1:8083_SOLR(ephemeral v=0)
> > >>>> >>>> MACHINE1:8082_SOLR(ephemeral v=0)
> > >>>> >>>> MACHINE1:8081_SOLR(ephemeral v=0)
> > >>>> >>>> MACHINE1:8084_SOLR(ephemeral v=0)
> > >>>> >>>> /COLLECTIONS (v=1 children=1)
> > >>>> >>>> COLLECTION1(v=0 children=2)"{"configName":"configuration1"}"
> > >>>> >>>> LEADER_ELECT(v=0 children=2)
> > >>>> >>>> SHARD1(V=0 children=1)
> > >>>> >>>> ELECTION(v=0 children=2)
> > >>>> >>>>
> > >>>> >>>> 87186203314552835-MACHINE1:8081_SOLR_-N_0000000096(ephemeral v=0)
> > >>>> >>>>
> > >>>> >>>> 87186203314552836-MACHINE1:8083_SOLR_-N_0000000084(ephemeral v=0)
> > >>>> >>>> SHARD2(v=0 children=1)
> > >>>> >>>> ELECTION(v=0 children=2)
> > >>>> >>>>
> > >>>> >>>> 231301391392833539-MACHINE1:8084_SOLR_-N_0000000085(ephemeral
> > v=0)
> > >>>> >>>>
> > >>>> >>>> 159243797356740611-MACHINE1:8082_SOLR_-N_0000000084(ephemeral
> > v=0)
> > >>>> >>>> LEADERS (v=0 children=2)
> > >>>> >>>> SHARD1 (ephemeral
> > >>>> >>>> v=0)"{"core":"","node_name":"MACHINE1:8081_solr","base_url":"
> > >>>> >>>> http://MACHINE1:8081/solr"}";
> > >>>> >>>> SHARD2 (ephemeral
> > >>>> >>>> v=0)"{"core":"","node_name":"MACHINE1:8082_solr","base_url":"
> > >>>> >>>> http://MACHINE1:8082/solr"}";
> > >>>> >>>> /OVERSEER_ELECT (v=0 children=2)
> > >>>> >>>> ELECTION (v=0 children=4)
> > >>>> >>>>
> > >>>> 231301391392833539-MACHINE1:8084_SOLR_-N_0000000251(ephemeral
> > >>>> >>> v=0)
> > >>>> >>>>
> > 87186203314552835-MACHINE1:8081_SOLR_-N_0000000248(ephemeral
> > >>>> >>> v=0)
> > >>>> >>>>
> > >>>> 159243797356740611-MACHINE1:8082_SOLR_-N_0000000250(ephemeral
> > >>>> >>> v=0)
> > >>>> >>>>
> > 87186203314552836-MACHINE1:8083_SOLR_-N_0000000249(ephemeral
> > >>>> >>> v=0)
> > >>>> >>>> LEADER (emphemeral
> > >>>> >>>> v=0)"{"id":"87186203314552835-MACHINE1:8081_solr-n_000000248"}"
> > >>>> >>>>
> > >>>> >>>>
> > >>>> >>>>
> > >>>> >>>> On Mon, Feb 27, 2012 at 2:47 PM, Mark Miller <
> > >>>> markrmil...@gmail.com>
> > >>>> >>> wrote:
> > >>>> >>>>
> > >>>> >>>>>
> > >>>> >>>>> On Feb 27, 2012, at 2:22 PM, Matthew Parker wrote:
> > >>>> >>>>>
> > >>>> >>>>>> Thanks for your reply Mark.
> > >>>> >>>>>>
> > >>>> >>>>>> I believe the build was towards the begining of the month. The
> > >>>> >>>>>> solr.spec.version is 4.0.0.2012.01.10.38.09
> > >>>> >>>>>>
> > >>>> >>>>>> I cannot access the clusterstate.json contents. I clicked on
> > it a
> > >>>> >>> couple
> > >>>> >>>>> of
> > >>>> >>>>>> times, but nothing happens. Is that stored on disk somewhere?
> > >>>> >>>>>
> > >>>> >>>>> Are you using the new admin UI? That has recently been updated
> > to
> > >>>> work
> > >>>> >>>>> better with cloud - it had some troubles not too long ago. If
> > you
> > >>>> are,
> > >>>> >>> you
> > >>>> >>>>> should trying using the old admin UI's zookeeper page - that
> > >>>> should
> > >>>> >>> show
> > >>>> >>>>> the cluster state.
> > >>>> >>>>>
> > >>>> >>>>> That being said, there has been a lot of bug fixes over the past
> > >>>> month
> > >>>> >>> -
> > >>>> >>>>> so you may just want to update to a recent version.
> > >>>> >>>>>
> > >>>> >>>>>>
> > >>>> >>>>>> I configured a custom request handler to calculate an unique
> > >>>> document
> > >>>> >>> id
> > >>>> >>>>>> based on the file's url.
> > >>>> >>>>>>
> > >>>> >>>>>> On Mon, Feb 27, 2012 at 1:13 PM, Mark Miller <
> > >>>> markrmil...@gmail.com>
> > >>>> >>>>> wrote:
> > >>>> >>>>>>
> > >>>> >>>>>>> Hey Matt - is your build recent?
> > >>>> >>>>>>>
> > >>>> >>>>>>> Can you visit the cloud/zookeeper page in the admin and send
> > the
> > >>>> >>>>> contents
> > >>>> >>>>>>> of the clusterstate.json node?
> > >>>> >>>>>>>
> > >>>> >>>>>>> Are you using a custom index chain or anything out of the
> > >>>> ordinary?
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>> - Mark
> > >>>> >>>>>>>
> > >>>> >>>>>>> On Feb 27, 2012, at 12:26 PM, Matthew Parker wrote:
> > >>>> >>>>>>>
> > >>>> >>>>>>>> TWIMC:
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> Environment
> > >>>> >>>>>>>> =========
> > >>>> >>>>>>>> Apache SOLR rev-1236154
> > >>>> >>>>>>>> Apache Zookeeper 3.3.4
> > >>>> >>>>>>>> Windows 7
> > >>>> >>>>>>>> JDK 1.6.0_23.b05
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> I have built a SOLR Cloud instance with 4 nodes using the
> > >>>> embeded
> > >>>> >>> Jetty
> > >>>> >>>>>>>> servers.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> I created a 3 node zookeeper ensemble to manage the solr
> > >>>> >>> configuration
> > >>>> >>>>>>> data.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> All the instances run on one server so I've had to move ports
> > >>>> around
> > >>>> >>>>> for
> > >>>> >>>>>>>> the various applications.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> I start the 3 zookeeper nodes.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> I started the first instance of solr cloud with the parameter
> > >>>> to
> > >>>> >>> have
> > >>>> >>>>> two
> > >>>> >>>>>>>> shards.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> The start the remaining 3 solr nodes.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> The system comes up fine. No errors thrown.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> I can view the solr cloud console and I can see the SOLR
> > >>>> >>> configuration
> > >>>> >>>>>>>> files managed by ZooKeeper.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> I published data into the SOLR Cloud instances from
> > SharePoint
> > >>>> using
> > >>>> >>>>>>> Apache
> > >>>> >>>>>>>> Manifold 0.4-incubating. Manifold is setup to publish the
> > data
> > >>>> into
> > >>>> >>>>>>>> collection1, which is the only collection defined in the
> > >>>> cluster.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> When I query the data from collection1 as per the solr wiki,
> > >>>> the
> > >>>> >>>>> results
> > >>>> >>>>>>>> are inconsistent. Sometimes all the results are there, other
> > >>>> times
> > >>>> >>>>>>> nothing
> > >>>> >>>>>>>> comes back at all.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> It seems to be having an issue auto replicating the data
> > >>>> across the
> > >>>> >>>>>>> cloud.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> Is there some specific setting I might have missed? Based
> > upon
> > >>>> what
> > >>>> >>> I
> > >>>> >>>>>>> read,
> > >>>> >>>>>>>> I thought that SOLR cloud would take care of distributing and
> > >>>> >>>>> replicating
> > >>>> >>>>>>>> the data automatically. Do you have to tell it what shard to
> > >>>> publish
> > >>>> >>>>> the
> > >>>> >>>>>>>> data into as well?
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> Any help would be appreciated.
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> Thanks,
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> Matt
> > >>>> >>>>>>>>
> > >>>> >>>>>>>> ------------------------------
> > >>>> >>>>>>>> This e-mail and any files transmitted with it may be
> > >>>> proprietary.
> > >>>> >>>>>>> Please note that any views or opinions presented in this
> > e-mail
> > >>>> are
> > >>>> >>>>> solely
> > >>>> >>>>>>> those of the author and do not necessarily represent those of
> > >>>> Apogee
> > >>>> >>>>>>> Integration.
> > >>>> >>>>>>>
> > >>>> >>>>>>> - Mark Miller
> > >>>> >>>>>>> lucidimagination.com
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>>
> > >>>> >>>>>>
> > >>>> >>>>>> Matt
> > >>>> >>>>>>
> > >>>> >>>>>> ------------------------------
> > >>>> >>>>>> This e-mail and any files transmitted with it may be
> > proprietary.
> > >>>> >>>>> Please note that any views or opinions presented in this e-mail
> > >>>> are
> > >>>> >>> solely
> > >>>> >>>>> those of the author and do not necessarily represent those of
> > >>>> Apogee
> > >>>> >>>>> Integration.
> > >>>> >>>>>
> > >>>> >>>>> - Mark Miller
> > >>>> >>>>> lucidimagination.com
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>>
> > >>>> >>>>
> > >>>> >>>> ------------------------------
> > >>>> >>>> This e-mail and any files transmitted with it may be proprietary.
> > >>>> >>> Please note that any views or opinions presented in this e-mail
> > are
> > >>>> solely
> > >>>> >>> those of the author and do not necessarily represent those of
> > Apogee
> > >>>> >>> Integration.
> > >>>> >>>
> > >>>> >>> - Mark Miller
> > >>>> >>> lucidimagination.com
> > >>>> >>>
> > >>>> >>>
> > >>>> >>>
> > >>>> >>>
> > >>>> >>>
> > >>>> >>>
> > >>>> >>>
> > >>>> >>>
> > >>>> >>>
> > >>>> >>>
> > >>>> >>>
> > >>>> >>>
> > >>>> >>
> > >>>> >
> > >>>> > ------------------------------
> > >>>> > This e-mail and any files transmitted with it may be proprietary.
> > >>>> Please note that any views or opinions presented in this e-mail are
> > solely
> > >>>> those of the author and do not necessarily represent those of Apogee
> > >>>> Integration.
> > >>>>
> > >>>> - Mark Miller
> > >>>> lucidimagination.com
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> >
> > ------------------------------
> > This e-mail and any files transmitted with it may be proprietary. Please
> > note that any views or opinions presented in this e-mail are solely those
> > of the author and do not necessarily represent those of Apogee Integration.
> >
>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
> ------------------------------
> This e-mail and any files transmitted with it may be proprietary. Please
> note that any views or opinions presented in this e-mail are solely those of
> the author and do not necessarily represent those of Apogee Integration.
>
>
- Mark Miller
lucidimagination.com
