Re: [sqlite] Re: building a custom DBD::SQLite

[P Kishor]

and therein lies the tale of computer programmers and users... Scott,
and others... I *am* good at a few things, but compiling programs is
not one of them. I am basically a copy artist -- I can follow
instructions, but the instructions have to be good and explicit. And
then, I can create new good things.

A good computer programmer will write instructions that will be clear
to good computer programmers, but may make little sense to folks such
as me. On the other hand, I may state things that are obvious to me,
but may have missed things important to a good computer programmer.

While I have been programming for almost half a decade now, I don't
consider myself to be a good computer programmer. I have come a long
way from the days of ColdFusion, and now I use Perl/JavaScript almost
exclusively, but I have miles to go when it comes to C :-).
Unfortunately, there are other things to learn along the way as well,
and they all take time.

On Nov 24, 2007 5:36 PM, Scott Hess <[EMAIL PROTECTED]> wrote:
> On Nov 18, 2007 3:28 PM, P Kishor <[EMAIL PROTECTED]> wrote:
> > Following those pretty much to the t, I almost got everything working.
> > Except, I got the following during make
> >
> > ../sqlite-3.5.2/ext/fts1/fts1.c:7:2: error: #error fts1 has a design
> > flaw and has been deprecated.
> > make: *** [fts1.lo] Error 1
> >
> > Fantastic. If it is has a design flaw and has been deprecated, then
> > why include it?
>
> In case someone is using it and really needs to keep using it and has
> reason to believe that the design flaw doesn't impact them.
>
> > Or, if it is included, where is the information that
> > it has been deprecated that one can read *before* doing make?
>
> That information is available on the mailing list - or wasn't that
> what you meant?  :-).

No, that is not what I meant. I would expect the instructions to be in
the Makefile.in, but it was already made clear that the Makefile was
not ready with fts instructions.
 very clearly says
"(Sorry - there's no makefile checked in yet. Coming soon.)"

I downloaded the program from sqlite.org. The most obvious place I
expected the instructions to be in was right in the download. That is
contextually the most relevant place to put the instructions. There is
a file called README in every fts folder in the src tree. I would
expect the most detailed instructions to be right there. They weren't
there.

>
> There isn't an obvious place to put such information, and fts1 and 2
> (and 3) are in any case considered outside the sqlite core.  It takes
> less than a minute to hit the error, so it's not like you lost three
> days tracking down some obscure design flaw yourself...

Yes, there is an obvious place, well, obvious to me. Obviously it
wasn't obvious to whoever put the instructions on the mailing list.
No, I did spend three days. Is that because I am stupid? Perhaps...
when it comes to compiling C programs. I am learning.

I spent a goodly amount of time just trying to figure out what to do
with the fts source. Since I couldn't find the exact instructions,
Google was my friend. I discovered Joe Wilson's fairly detailed
description on the mailing list (see
).
This is what should have been on sqlite.org to begin with. Want to
something silly? Joe provided a patch to the Makefile, but I don't
even know how to apply a patch. Yes, to someone who doesn't know these
things, they are all mysteries. So, I basically eyeballed each line
and hand edited my Makefile.

Well, one line in Joe's patch looks like so

+# FTS2 (optional)
+LIBOBJ += fts2.lo fts2_hash.lo fts2_porter.lo fts2_tokenizer1.lo

I followed the above, but look above... fts2_tokenizer.lo is missing
from the line above. Since I followed the above blindly, I started
getting the missing symbol error.

Between that and the -flat_namespace issue, three days were gone poof!
The flat namespace issue is documented on developer.apple.com, but
again, the instructions to solve it are not clear. I understand the
issue conceptually, but I don't know the exact ingredients and
instructions to solve it.

..

An extensive back and forth with a local Madison friend who knows more
than I do re. compiling programs on a Mac revealed that I needed to
add the MACOSX_DEPLOYMENT_TARGET. Then he discovered that I was
missing fts2_tokenizer.lo

Added that, and boom! I was in business... three days later.

I believe that the instructions I have added at
 are detailed and
descriptive enough that someone with my level of knowledge (which is
very little) will be able to compile fts. Now when I Google for
"compiling fts sqlite" my page comes up first, and a n00b like me will
be able to be on his/her way quickly.

We spent a lot of back-and-forth on how to make the sqlite.org website
better, but I am not sure it is any better. It looks pretty much the
same as it did. 

Re: [sqlite] Re: building a custom DBD::SQLite

[Scott Hess]

On Nov 18, 2007 3:28 PM, P Kishor <[EMAIL PROTECTED]> wrote:
> Following those pretty much to the t, I almost got everything working.
> Except, I got the following during make
>
> ../sqlite-3.5.2/ext/fts1/fts1.c:7:2: error: #error fts1 has a design
> flaw and has been deprecated.
> make: *** [fts1.lo] Error 1
>
> Fantastic. If it is has a design flaw and has been deprecated, then
> why include it?

In case someone is using it and really needs to keep using it and has
reason to believe that the design flaw doesn't impact them.

> Or, if it is included, where is the information that
> it has been deprecated that one can read *before* doing make?

That information is available on the mailing list - or wasn't that
what you meant?  :-).

There isn't an obvious place to put such information, and fts1 and 2
(and 3) are in any case considered outside the sqlite core.  It takes
less than a minute to hit the error, so it's not like you lost three
days tracking down some obscure design flaw yourself...

> Anyway, I went back into Makefile.in and commented out all references
> to fts1 and ran make again. This time I got
>
> ../sqlite-3.5.2/ext/fts2/fts2.c:7:2: error: #error fts2 has a design
> flaw and has been deprecated.
> make: *** [fts2.lo] Error 1
>
> So, I went back into the ext library and discovered that there was
> fts3.

Note that the comment just before the SQLITE_ENABLE_BROKEN_FTS1
snippet explicitely states that you should be using fts3 or higher.

> So, I went back into Makefile.in and commented out all
> references to fts2, added similar looking lines for fts3 and ran make
> again. This time it all worked. Great!
>
> Well, not so great. Now when I run the new sqlite3, I get the following
>
> dyld: lazy symbol binding failed: Symbol not found: _sqlite3Fts1Init
>   Referenced from:
> /Users/punkish/Projects/sqlite-3.5.2-build/.libs/libsqlite3.0.dylib
>   Expected in: flat namespace
>
> dyld: Symbol not found: _sqlite3Fts1Init
>   Referenced from:
> /Users/punkish/Projects/sqlite-3.5.2-build/.libs/libsqlite3.0.dylib
>   Expected in: flat namespace
>
> Trace/BPT trap

The lines I have in main.mk for fts3 stuff look like:

# Near the top after "Object files for the SQLite library".
LIBOBJ += fts3.o \
  fts3_hash.o \
  fts3_icu.o \
  fts3_porter.o \
  fts3_tokenizer.o \
  fts3_tokenizer1.o

# After TESTSRC definition.
TESTSRC += $(TOP)/ext/fts3/fts3_tokenizer.c

# After current "Rules to build the extension objects." section
fts3.o: $(TOP)/ext/fts3/fts3.c $(HDR) $(EXTHDR)
$(TCCX) -DSQLITE_CORE -c $(TOP)/ext/fts3/fts3.c

fts3_hash.o:$(TOP)/ext/fts3/fts3_hash.c $(HDR) $(EXTHDR)
$(TCCX) -DSQLITE_CORE -c $(TOP)/ext/fts3/fts3_hash.c

fts3_icu.o: $(TOP)/ext/fts3/fts3_icu.c $(HDR) $(EXTHDR)
$(TCCX) -DSQLITE_CORE -c $(TOP)/ext/fts3/fts3_icu.c

fts3_porter.o:  $(TOP)/ext/fts3/fts3_porter.c $(HDR) $(EXTHDR)
$(TCCX) -DSQLITE_CORE -c $(TOP)/ext/fts3/fts3_porter.c

fts3_tokenizer.o:   $(TOP)/ext/fts3/fts3_tokenizer.c $(HDR) $(EXTHDR)
$(TCCX) -DSQLITE_CORE -c $(TOP)/ext/fts3/fts3_tokenizer.c

fts3_tokenizer1.o:  $(TOP)/ext/fts3/fts3_tokenizer1.c $(HDR) $(EXTHDR)
$(TCCX) -DSQLITE_CORE -c $(TOP)/ext/fts3/fts3_tokenizer1.c

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Re: building a custom DBD::SQLite

[Joe Wilson]

Grep for the unresolved external in all the *.[ch] files.
sqlite3Fts3InitHashTable lives in ext/fts3/fts3_tokenizer.c.
Make sure it is in your makefile.

Ditto for fts2, with a slightly different name.

If you're building fts2 make sure you compile with 

  -DSQLITE_ENABLE_BROKEN_FTS2
  -DSQLITE_ENABLE_FTS2
  -DSQLITE_CORE

When In doubt, grep and read the source files.
There's a lot of info there.

--- P Kishor <[EMAIL PROTECTED]> wrote:
> Here are my travails thus far. I have been singularly unable to build
> SQLite 3.5.2 with fts3 on Intel Mac OS X 10.4.11
> 
> I added the following to my Makefile.in
> 
> #TCC += -DSQLITE_OMIT_LOAD_EXTENSION=1
> TCC += -DSQLITE_CORE=1
> TCC += -DSQLITE_ENABLE_FTS3=1
> 
> and then added all the fts3 .c and .h files in the appropriate places
> as documented by Joe Wilson in another email. Then I did the following
> 
> $ export MACOSX_DEPLOYMENT_TARGET=10.3
> $ ./configure
> $ make
> 
> make croaks with the following error
> 
> /usr/bin/ld: Undefined symbols:
> _sqlite3Fts3InitHashTable
> collect2: ld returned 1 exit status
> make: *** [sqlite3] Error 1



  

Get easy, one-click access to your favorites. 
Make Yahoo! your homepage.
http://www.yahoo.com/r/hs 

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

[sqlite] Re: building a custom DBD::SQLite

[P Kishor]

Here are my travails thus far. I have been singularly unable to build
SQLite 3.5.2 with fts3 on Intel Mac OS X 10.4.11

I added the following to my Makefile.in

#TCC += -DSQLITE_OMIT_LOAD_EXTENSION=1
TCC += -DSQLITE_CORE=1
TCC += -DSQLITE_ENABLE_FTS3=1

and then added all the fts3 .c and .h files in the appropriate places
as documented by Joe Wilson in another email. Then I did the following

$ export MACOSX_DEPLOYMENT_TARGET=10.3
$ ./configure
$ make

make croaks with the following error

/usr/bin/ld: Undefined symbols:
_sqlite3Fts3InitHashTable
collect2: ld returned 1 exit status
make: *** [sqlite3] Error 1

This train is not going anywhere at all. In another post, my
experience with fts2

On Nov 18, 2007 5:28 PM, P Kishor <[EMAIL PROTECTED]> wrote:
>
> On Nov 18, 2007 2:59 PM, P Kishor <[EMAIL PROTECTED]> wrote:
> > On Nov 17, 2007 2:22 PM, P Kishor <[EMAIL PROTECTED]> wrote:
> > > I need to build a DBD::SQLite package using SQLite with fts
> > > capabilities. Can someone on this list kindly give me painless, easy
> > > to understand instructions to do so? I need to do this on a Mac OS X
> > > 10.4 first, and then on my web host subsequently (running some flavor
> > > of Linux, natch).
> > >
> >
> >
> > ok, let's try this again.
> >
> > I found instructions for building a loadable extension at
> > 
> >
> > Note to Richard on the subject of improving the website -- please make
> > such instructions easier to find by perhaps linking them somewhere
> > prominently under "building or compiling" right off the download page
> > for the source code.
> >
> > That said, I tried to build the fts2 extension and got the following --
> >
> > $ tar xvzf sqlite-3.4.2
> > $ mkdir sqlite-3.4.2-build
> > $ cd sqlite-3.4.2
> > 
> > 
> > $ cd ..
> > $ cd sqlite-3.4.2-build
> > $ ../sqlite-3.4.2/configure LIBS=-ldl
> > $ make
> > $ export LD_LIBRARY_PATH="`pwd`:$LD_LIBRARY_PATH"
> > $ gcc -I`pwd` -shared ../sqlite-3.4.2/ext/fts2/fts2.c -o fts2.so
> > i686-apple-darwin8-gcc-4.0.1: unrecognized option '-shared'
> > /usr/bin/ld: Undefined symbols:
> > _main
> > _sqlite3Fts2HashClear
> > _sqlite3Fts2HashFind
> > _sqlite3Fts2HashInit
> > _sqlite3Fts2HashInsert
> > _sqlite3Fts2InitHashTable
> > _sqlite3Fts2PorterTokenizerModule
> > _sqlite3Fts2SimpleTokenizerModule
> > collect2: ld returned 1 exit status
> >
> >
> > so, this is the first step that I have surmount. Once this is done, I
> > would really like fts2 to not be shared but be permanently jammed into
> > sqlite3 so I don't have to load it manually (unless, there is any
> > significant advantage to building a shared library).
>
> this is getting more and more a happy waste of time (hopefully I would
> have learned something out of this). After much searching, I found Joe
> Wilson's instructions at
> 
>
> Following those pretty much to the t, I almost got everything working.
> Except, I got the following during make
>
> ../sqlite-3.5.2/ext/fts1/fts1.c:7:2: error: #error fts1 has a design
> flaw and has been deprecated.
> make: *** [fts1.lo] Error 1
>
> Fantastic. If it is has a design flaw and has been deprecated, then
> why include it? Or, if it is included, where is the information that
> it has been deprecated that one can read *before* doing make?
>
> Anyway, I went back into Makefile.in and commented out all references
> to fts1 and ran make again. This time I got
>
> ../sqlite-3.5.2/ext/fts2/fts2.c:7:2: error: #error fts2 has a design
> flaw and has been deprecated.
> make: *** [fts2.lo] Error 1
>
> So, I went back into the ext library and discovered that there was
> fts3. So, I went back into Makefile.in and commented out all
> references to fts2, added similar looking lines for fts3 and ran make
> again. This time it all worked. Great!
>
> Well, not so great. Now when I run the new sqlite3, I get the following
>
> dyld: lazy symbol binding failed: Symbol not found: _sqlite3Fts1Init
>   Referenced from:
> /Users/punkish/Projects/sqlite-3.5.2-build/.libs/libsqlite3.0.dylib
>   Expected in: flat namespace
>
> dyld: Symbol not found: _sqlite3Fts1Init
>   Referenced from:
> /Users/punkish/Projects/sqlite-3.5.2-build/.libs/libsqlite3.0.dylib
>   Expected in: flat namespace
>
> Trace/BPT trap
>
> If I can get some clear instructions on how to do this successfully, I
> promise to record them on the wiki for other poor souls who might try
> this.
>
> Many thanks in advance.
>
> And, oh yes, still will need help with the following once the above is 
> possible.
>
>
>
> >
> > Once I am successful with the above, I would like to build a
> > DBD::SQLite with my new library.
> >
> > Many thanks for your guidance.
> >
> > --
> > Puneet Kishor
> >
>

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

[sqlite] Re: building a custom DBD::SQLite

[P Kishor]

On Nov 18, 2007 2:59 PM, P Kishor <[EMAIL PROTECTED]> wrote:
> On Nov 17, 2007 2:22 PM, P Kishor <[EMAIL PROTECTED]> wrote:
> > I need to build a DBD::SQLite package using SQLite with fts
> > capabilities. Can someone on this list kindly give me painless, easy
> > to understand instructions to do so? I need to do this on a Mac OS X
> > 10.4 first, and then on my web host subsequently (running some flavor
> > of Linux, natch).
> >
>
>
> ok, let's try this again.
>
> I found instructions for building a loadable extension at
> 
>
> Note to Richard on the subject of improving the website -- please make
> such instructions easier to find by perhaps linking them somewhere
> prominently under "building or compiling" right off the download page
> for the source code.
>
> That said, I tried to build the fts2 extension and got the following --
>
> $ tar xvzf sqlite-3.4.2
> $ mkdir sqlite-3.4.2-build
> $ cd sqlite-3.4.2
> 
> 
> $ cd ..
> $ cd sqlite-3.4.2-build
> $ ../sqlite-3.4.2/configure LIBS=-ldl
> $ make
> $ export LD_LIBRARY_PATH="`pwd`:$LD_LIBRARY_PATH"
> $ gcc -I`pwd` -shared ../sqlite-3.4.2/ext/fts2/fts2.c -o fts2.so
> i686-apple-darwin8-gcc-4.0.1: unrecognized option '-shared'
> /usr/bin/ld: Undefined symbols:
> _main
> _sqlite3Fts2HashClear
> _sqlite3Fts2HashFind
> _sqlite3Fts2HashInit
> _sqlite3Fts2HashInsert
> _sqlite3Fts2InitHashTable
> _sqlite3Fts2PorterTokenizerModule
> _sqlite3Fts2SimpleTokenizerModule
> collect2: ld returned 1 exit status
>
>
> so, this is the first step that I have surmount. Once this is done, I
> would really like fts2 to not be shared but be permanently jammed into
> sqlite3 so I don't have to load it manually (unless, there is any
> significant advantage to building a shared library).

this is getting more and more a happy waste of time (hopefully I would
have learned something out of this). After much searching, I found Joe
Wilson's instructions at


Following those pretty much to the t, I almost got everything working.
Except, I got the following during make

../sqlite-3.5.2/ext/fts1/fts1.c:7:2: error: #error fts1 has a design
flaw and has been deprecated.
make: *** [fts1.lo] Error 1

Fantastic. If it is has a design flaw and has been deprecated, then
why include it? Or, if it is included, where is the information that
it has been deprecated that one can read *before* doing make?

Anyway, I went back into Makefile.in and commented out all references
to fts1 and ran make again. This time I got

../sqlite-3.5.2/ext/fts2/fts2.c:7:2: error: #error fts2 has a design
flaw and has been deprecated.
make: *** [fts2.lo] Error 1

So, I went back into the ext library and discovered that there was
fts3. So, I went back into Makefile.in and commented out all
references to fts2, added similar looking lines for fts3 and ran make
again. This time it all worked. Great!

Well, not so great. Now when I run the new sqlite3, I get the following

dyld: lazy symbol binding failed: Symbol not found: _sqlite3Fts1Init
  Referenced from:
/Users/punkish/Projects/sqlite-3.5.2-build/.libs/libsqlite3.0.dylib
  Expected in: flat namespace

dyld: Symbol not found: _sqlite3Fts1Init
  Referenced from:
/Users/punkish/Projects/sqlite-3.5.2-build/.libs/libsqlite3.0.dylib
  Expected in: flat namespace

Trace/BPT trap

If I can get some clear instructions on how to do this successfully, I
promise to record them on the wiki for other poor souls who might try
this.

Many thanks in advance.

And, oh yes, still will need help with the following once the above is possible.


>
> Once I am successful with the above, I would like to build a
> DBD::SQLite with my new library.
>
> Many thanks for your guidance.
>
> --
> Puneet Kishor
>

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

[sqlite] Re: building a custom DBD::SQLite

[P Kishor]

On Nov 17, 2007 2:22 PM, P Kishor <[EMAIL PROTECTED]> wrote:
> I need to build a DBD::SQLite package using SQLite with fts
> capabilities. Can someone on this list kindly give me painless, easy
> to understand instructions to do so? I need to do this on a Mac OS X
> 10.4 first, and then on my web host subsequently (running some flavor
> of Linux, natch).
>


ok, let's try this again.

I found instructions for building a loadable extension at


Note to Richard on the subject of improving the website -- please make
such instructions easier to find by perhaps linking them somewhere
prominently under "building or compiling" right off the download page
for the source code.

That said, I tried to build the fts2 extension and got the following --

$ tar xvzf sqlite-3.4.2
$ mkdir sqlite-3.4.2-build
$ cd sqlite-3.4.2


$ cd ..
$ cd sqlite-3.4.2-build
$ ../sqlite-3.4.2/configure LIBS=-ldl
$ make
$ export LD_LIBRARY_PATH="`pwd`:$LD_LIBRARY_PATH"
$ gcc -I`pwd` -shared ../sqlite-3.4.2/ext/fts2/fts2.c -o fts2.so
i686-apple-darwin8-gcc-4.0.1: unrecognized option '-shared'
/usr/bin/ld: Undefined symbols:
_main
_sqlite3Fts2HashClear
_sqlite3Fts2HashFind
_sqlite3Fts2HashInit
_sqlite3Fts2HashInsert
_sqlite3Fts2InitHashTable
_sqlite3Fts2PorterTokenizerModule
_sqlite3Fts2SimpleTokenizerModule
collect2: ld returned 1 exit status


so, this is the first step that I have surmount. Once this is done, I
would really like fts2 to not be shared but be permanently jammed into
sqlite3 so I don't have to load it manually (unless, there is any
significant advantage to building a shared library).

Once I am successful with the above, I would like to build a
DBD::SQLite with my new library.

Many thanks for your guidance.

--
Puneet Kishor

-
To unsubscribe, send email to [EMAIL PROTECTED]
-