Re: failed to store index data

Sun, 19 Mar 2023 12:45:08 -0700 




From your previous emails, you set real-time (rt) indexing. For such 
setup,

don't run the indexer utility, no need for it any longer.

Also be sure to update the pilersearch.service as in
https://bitbucket.org/jsuto/piler/src/master/systemd/pilersearch.service

I think you'll be fine after that, manticore will create the required
index files.

Janos

On 2023-03-19 20:36, Jonathan Baecker wrote:

This folder is empty and I also can not reindex. I recover my system
to a state where it was works. Maybe I miss a step, so here are my
steps:

1. piler 1.4.3 runs with sphinx under debian 11.
2. I install manticore 6.0.x from webpage
3. cp /usr/local/etc/piler/manticore.conf.dist
/usr/local/etc/piler/manticore.conf
4. add db settings to /usr/local/etc/piler/manticore.conf
4. apply all changes from mailpiler doc about switching to manticore
5. when I run as piler user: indexer --all --config
/usr/local/etc/piler/manticore.conf it gives me: FATAL: no tables
found in config file '/usr/local/etc/piler/manticore.conf'
6. I reboot the system to have all services running

systemctl status pilersearch.service show:

    Process: 611 ExecStart=/usr/bin/searchd --config
/usr/local/etc/piler/sphinx.conf (code=exited, status=0/SUCCESS)
   Main PID: 615 (searchd)
      Tasks: 10 (limit: 4675)
     Memory: 52.2M
        CPU: 314ms
     CGroup: /system.slice/pilersearch.service

             ├─614 /usr/bin/searchd --config 
/usr/local/etc/piler/sphinx.conf
             ├─615 /usr/bin/searchd --config 
/usr/local/etc/piler/sphinx.conf

             └─620 /usr/bin/manticore-executor -n
/tmp/manticore-buddy.phar --listen=http://127.0.0.1:9312 --threads=2

Mär 19 20:27:40 piler searchd[611]: Copyright (c) 2008-2016, Sphinx
Technologies Inc (http://sphinxsearch.com)
Mär 19 20:27:40 piler searchd[611]: Copyright (c) 2017-2023, Manticore
Software LTD (https://manticoresearch.com)
Mär 19 20:27:40 piler searchd[611]: WARNING: key 'docinfo' was
permanently removed from configuration. Refer to documentation for
details.

So it still uses the old sphinx.conf

Was my steps ok?



Am 19.03.23 um 18:48 schrieb Janos SUTO:



Your configs look fine. The errno=1064 means that no such local index.
Show me "ls -la /var/piler/manticore/" output, too.

In my test environment it looks like:

ls -la /var/piler/manticore/
total 4796
drwx------  2 piler piler    4096 Mar 19 17:43 .
drwxr-xr-x 10 root  root     4096 Mar 19 17:01 ..
-rw-------  1 piler piler       8 Mar 19 17:43 binlog.001
-rw-------  1 piler piler       0 Mar 19 17:03 binlog.lock
-rw-------  1 piler piler      11 Mar 19 17:43 binlog.meta
-rw-------  1 piler piler    2731 Mar 19 17:43 manticore.log
-rw-------  1 piler piler       0 Mar 19 17:03 note1.lock
-rw-------  1 piler piler    1200 Mar 19 17:03 note1.meta
-rw-------  1 piler piler      20 Mar 19 17:03 note1.ram
-rw-------  1 piler piler       0 Mar 19 17:03 piler1.lock
-rw-------  1 piler piler    1569 Mar 19 17:43 piler1.meta
-rw-------  1 piler piler 4862221 Mar 19 17:43 piler1.ram
-rw-------  1 piler piler     893 Mar 19 17:01 query.log
-rw-------  1 piler piler       0 Mar 19 17:03 tag1.lock
-rw-------  1 piler piler    1199 Mar 19 17:03 tag1.meta
-rw-------  1 piler piler      20 Mar 19 17:03 tag1.ram

Janos

On 2023-03-19 17:15, Jonathan Baecker wrote:

Ah sorry! Here is the output, but I run: 
/usr/local/etc/piler/manticore.conf


#
# minimal manticore configuration suited to piler
#


index piler1
{
    type = rt
    path = /var/piler/manticore/piler1
    rt_mem_limit = 512M
    stored_fields =
    min_word_len = 1
    min_prefix_len = 5
    charset_table  = 0..9, english, _, \
                                  U+C1->U+E1, U+C4->U+E4, U+C5->U+E5,
U+C6->U+E6, U+C9->U+E9, U+CD->U+ED, U+D3->U+F3, U+D6->U+F6,
U+D8->U+F8, \
                                  U+DA->U+FA, U+DC->U+FC,
U+0150->U+0151, U+0152->U+0153, U+0170->U+0171, U+01E2->U+E6,
U+01E3->U+E6, U+01FC->U+E6, \
                                  U+01FD->U+E6, U+1D01->U+E6,
U+1D02->U+E6, U+1D2D->U+E6, U+1D46->U+E6, \
                                  U+DF, U+E1, U+E4, U+E5, U+E6, U+E9,
U+ED, U+00F3, U+F6, U+F8, U+FA, U+FC, U+0151, U+0153, U+0171
    # See

https://manual.manticoresearch.com/Creating_an_index/Data_types#Row-wise-and-columnar-attribute-storages 
    # if you want to enable columnar storage

    # columnar_attrs = *
    rt_field = sender
    rt_field = rcpt
    rt_field = senderdomain
    rt_field = rcptdomain
    rt_field = subject
    rt_field = body
    rt_field = attachment_types
    rt_attr_bigint = arrived
    rt_attr_bigint = sent
    rt_attr_uint = size
    rt_attr_uint = direction
    rt_attr_uint = folder
    rt_attr_uint = attachments
}

index tag1
{
    type = rt
    path = /var/piler/manticore/tag1
    rt_mem_limit = 16M
    stored_fields = tag
    min_word_len = 2
    min_prefix_len = 5
    charset_table  = 0..9, english, _, \
                                  U+C1->U+E1, U+C4->U+E4, U+C5->U+E5,
U+C6->U+E6, U+C9->U+E9, U+CD->U+ED, U+D3->U+F3, U+D6->U+F6,
U+D8->U+F8, \
                                  U+DA->U+FA, U+DC->U+FC,
U+0150->U+0151, U+0152->U+0153, U+0170->U+0171, U+01E2->U+E6,
U+01E3->U+E6, U+01FC->U+E6, \
                                  U+01FD->U+E6, U+1D01->U+E6,
U+1D02->U+E6, U+1D2D->U+E6, U+1D46->U+E6, \
                                  U+DF, U+E1, U+E4, U+E5, U+E6, U+E9,
U+ED, U+00F3, U+F6, U+F8, U+FA, U+FC, U+0151, U+0153, U+0171
    rt_field = tag
    rt_attr_bigint = mid
    rt_attr_uint = uid
}

index note1
{
    type = rt
    path = /var/piler/manticore/note1
    rt_mem_limit = 16M
    stored_fields = note
    min_word_len = 2
    min_prefix_len = 5
    charset_table  = 0..9, english, _, \
                                  U+C1->U+E1, U+C4->U+E4, U+C5->U+E5,
U+C6->U+E6, U+C9->U+E9, U+CD->U+ED, U+D3->U+F3, U+D6->U+F6,
U+D8->U+F8, \
                                  U+DA->U+FA, U+DC->U+FC,
U+0150->U+0151, U+0152->U+0153, U+0170->U+0171, U+01E2->U+E6,
U+01E3->U+E6, U+01FC->U+E6, \
                                  U+01FD->U+E6, U+1D01->U+E6,
U+1D02->U+E6, U+1D2D->U+E6, U+1D46->U+E6, \
                                  U+DF, U+E1, U+E4, U+E5, U+E6, U+E9,
U+ED, U+00F3, U+F6, U+F8, U+FA, U+FC, U+0151, U+0153, U+0171
    rt_field = note
    rt_attr_bigint = mid
    rt_attr_uint = uid
}

searchd
{
        listen                  = 127.0.0.1:9312
        listen                  = 127.0.0.1:9306:mysql41
        log                     = /var/piler/manticore/manticore.log
        binlog_max_log_size     = 256M
        binlog_path             = /var/piler/manticore
        binlog_flush            = 2
        query_log               = /var/piler/manticore/query.log
        network_timeout         = 5
        pid_file                = /var/run/piler/searchd.pid
        seamless_rotate         = 1
        preopen_indexes         = 1
        unlink_old              = 1
        thread_stack            = 512k
        rt_flush_period         = 300

}


Am 19.03.23 um 16:35 schrieb Janos SUTO:



I meant to execute /etc/piler/manticore.conf (yes, it's executable).

Janos

On 2023-03-19 16:06, Jonathan Baecker wrote:

Am 19.03.23 um 14:27 schrieb Janos SUTO:

Hello Jonathan,


Hello Jano, thank you for you quick response!


please tell me more about your setup. Show me the output of

pilerconf |grep -E '(sph|rtindex)'


Output is:

rtindex=1
sphxdb=piler1
sphxhost=127.0.0.1
sphxport=9306



and

/etc/piler/manticore.conf


Content is:

define('RT', 1);



Be sure to redact any passwords.

Janos


On 2023-03-19 14:11, Jonathan Baecker wrote:

Hello,

I just update piler from 1.3.9 to 1.4.3 and I also switched to

manticore. But now piler can not receive new mails. when I check 
piler

service, I get this error:


Mär 19 14:06:52 piler piler[567]:
50000000641708f61c491c9c0069477d77cb: hdr_len: 3130, offset: 3
Mär 19 14:06:52 piler piler[567]:
50000000641708f61c491c9c0069477d77cb: touch

2e5fa096927b9a6ac6fe82ff2b81256acfa719eb32031ca68f0d86db45415d9f 
OK

(<608c730b-8a9d-c4aa-4ebd-7f3dec012...@mail.de>)
Mär 19 14:06:52 piler piler[567]:
50000000641708f61c491c9c0069477d77cb: stored
'50000000641708f61c491c9c0069477d77cb.m' 39327/11168 bytes
Mär 19 14:06:52 piler piler[567]:
50000000641708f61c491c9c0069477d77cb: updated meta reference for
'<FxXfleYASv2VKckQYZ7mWg@geopod-ismtpd-36>', rc=0
Mär 19 14:06:52 piler piler[567]:
50000000641708f61c491c9c0069477d77cb: stored 1 recipients, rc=0
Mär 19 14:06:52 piler piler[567]: ERROR:

50000000641708f61c491c9c0069477d77cb failed to store index data 
for

id=31270, errno=1064, append ret=0
Mär 19 14:06:52 piler piler[567]: ERROR:

50000000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE 
FROM

sph_index WHERE id=31270
Mär 19 14:06:52 piler piler[567]: ERROR:

50000000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE 
FROM

rcpt WHERE id=31270
Mär 19 14:06:52 piler piler[567]: ERROR:

50000000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE 
FROM

metadata WHERE id=31270
Mär 19 14:06:52 piler piler[567]: ERROR:

50000000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE 
FROM

attachment WHERE piler_id='50000000641708f61c491c9c0069477d77cb'
Mär 19 14:06:52 piler piler[567]:
50000000641708f61c491c9c0069477d77cb: unlinking
/var/piler/store/00/641/77/cb/50000000641708f61c491c9c0069477d77cb.m
Mär 19 14:06:52 piler piler[567]:
50000000641708f61c491c9c0069477d77cb: stored metadata, rc=1
Mär 19 14:06:52 piler piler[567]: 1/E07M4E0192DW6TN0: moved to
/var/piler/error/E07M4E0192DW6TN0
Mär 19 14:06:52 piler piler[567]: 1/E07M4E0192DW6TN0:

50000000641708f61c491c9c0069477d77cb, size=39327/11168, 
attachments=0,

reference=<FxXfleYASv2VKckQYZ7mWg@geopod-ismtpd-36>,
message-id=<608c730b-8a9d-c4aa-4ebd-7f3dec012...@mail.de>,
retention=2557, folder=0, delay=0.1183, status=error



After Update I run mysql piler < ./util/db-upgrade.sql But I 
think it

had done nothing.

Do you have any idea, how to fix this?


Have a good day!

jb






























					Reply via email to