Re: pilerexport w flag matching

[sj]

Hello Ryan,

try getting the master branch, do a complete build, however, don't 
install it.
Instead enter the src directory, use LD_LIBRARY_PATH=. to make sure 
./pilerexport
will use the recently built libpiler.so file, and try again. Also it's
worth to check what's going on 127.0.0.1:9306 with ngrep or similar 
tool.

Janos

On 2021-04-08 08:45, Ryan Blenis wrote:
Hi Janos,

Thanks as always for the reply. No worries on the -w replacing the
other switches, a documentation update will clear that up pretty
easily, and thank you, I'd love a total in the dry run!

As for "the export utility assumes searchd is listening on
127.0.0.1:9306 [1] That part didn't change from 1.3.8 to 1.3.11."

I looked back at all the commits and didn't see anything in that file
that could be causing it, but I'll investigate because now it's just
annoying me that it doesn't work. I saw that it was assuming searchd
was on 127.0.0.1:9306 [1], and it is on my system, however it still
fails with that error in the new build. I don't know if you have a
test case for pilerexport -w, but it should be repeatable. I don't see
much changing in the cfg.c either, but when I printf the cfg.mysqluser
and cfg.mysqlpwd on the old (1.3.8) version I get the correct info. If
I printf cfg.mysqluser and cfg.mysqlpwd on the new version I get "r"
and "" respectively. A quick strace of pilerexport shows it opening
the correct conf file (/usr/local/etc/piler/piler.conf) on both
versions. And specifying the config file with -c did not help either.
Not sure if you can reproduce this on your end or not.

The odd part is, cfg.mysqluser and cfg.mysqlpwd _IS_ utilized in the
"init_session_data(&sdata, &cfg);" line above the
"init_session_data(&sdata2, &cfg);" and it passes the database "open"
test just fine there...

Thank you.

On Thu, Apr 8, 2021 at 12:13 AM <s...@acts.hu> wrote:

Hello Ryan,

On 2021-04-08 01:36, Ryan Blenis wrote:

Thanks, that led me to what is causing the issue / confusion.

The -w switch is described as "Where condition to pass to sphinx,
eg.
"match('@subject: piler')"

Which led me to believe the MATCH string was all that was supposed
to
be there/replaced, however a quick look at the code shows that if
-w
is used, it REPLACES the ENTIRE where clause. This distinction
means

yes, perhaps the "eg." was not that prominent in the short --help
output.
I'll improve the docs on the website, it's lagging behind the actual

features,
and I'll add a clarification on it.

The simplest workaround to this for others would be to note that
-w
allows you to build your own query and negates the use of other
parameters. The ideal fix I think would be to still utilize the
other
parameters, but have -w content appended within the MATCH()
portion of
the query.

Such fix would only complicate things because you can define the
whole
query using -w including the time frame, recipients, etc. Again,
I'll
add a clarification to the docs.

Aside from that: I realize I'm behind on piler (1.3.8), and would
like
to update to get the latest pilerexport with zip capabilities, yet
I
see there is no upgrade information on
https://www.mailpiler.org/wiki/current:upgrade . What is the
process
to the latest (1.3.11)?

Well, simply compile the new stuff, and overwrite the binaries and
the
GUI files. The database schema hasn't changed from 1.3.8 to 1.3.11.
However, don't rush with that. The zip export feature has a poor
performance that needs a rework.

I'd also like to add a "--num-only" type flag to pilerexport to
see
the number of matches before exporting (would probably imply
dryrun).

We have something similar. When specifying -d (or --dry-run) it
prints
the matching serial ids, eg.

$ pilerexport -d -w "MATCH(' some query')"
id:318
id:375
id:518
id:656
id:660
id:688
id:733

I'll improve it to add "total:7" as the last line of the output, if
that's OK.

When trying to just compile the latest, I get the error "error:
cannot
connect to 127.0.0.1:9306 [1] [1]" so I'm not sure if that's an
issue
because not all the components are upgraded, or if I had a
different
configure flag/path configured during the original install.

The export utility assumes searchd is listening on 127.0.0.1:9306
[1]
That part didn't change from 1.3.8 to 1.3.11.

Janos


Links:
------
[1] http://127.0.0.1:9306