Re: Messages wrong date

[sj]

Hello Marcelo,

Identify these emails in the metadata table by checking for the sent 
column.
Their values should be a timestamp for 2036 (which is clearly wrong and 
either
a parser issue or a malformed Date: header). Then fix their sent values 
to

the same as the arrived column.

Janos

On 2022-10-18 14:51, Marcelo Machado wrote:

Hello everyone.

How do I fix the date of messages that appear with the date of 2036.

Thanks in advance.

Marcelo Gomes

Re: fetching mail from office365 after Microsoft disables "basic authentication"?

[sj]

Hello Jeremiah,

I'll try to come up with something usable starting at here:
https://stackoverflow.com/questions/54189837/python-o365-authentication-when-basic-authentication-becomes-deprecated

Note that I'll prefer the imapfetch.py utility to get emails via imap.

Janos SUTO

On 2022-07-01 00:08, Jeremiah Poling wrote:

Hello!

We're pilerimport -i to download mail for archiving from a journaling
mailbox.

Are there any recommendations for a similar method that would use a
more "modern" authentication method for after Microsoft disables basic
authentication in September?

Thank you for any advice you can provide!

--Jeremiah

Re: MailPiler on FreeBSD

[sj]

Hello Hagen,

piler in the early 1.3.x days has switched to epoll which is a Linux 
only method.
I'm not sure if the FreeBSD Linux-compatibility mode actually has proper 
support
to run such binaries. I'd be glad if you could try and share your 
experiences.


Anyway, you are right, the prefork branch hasn't been updated for a long 
time, and

I have no plan to do so.

Janos

On 2022-03-17 10:15, Hagen Bauer wrote:

Hi to all,

I am planing to move my MailPiler Server from a Debian Server to a 
FreeBSD Jail.


According to the documenation MailPiler is supported on FreeBSD

If I read this sentence

For FreeBSD you must download (or checkout) the prefork branch


and look into the branch at 
https://bitbucket.org/jsuto/piler/src/prefork/src/


I get the impression that this branch is not at the newest version.

Did I missed something or made a mistake in my research?

Is FreeBSD a plattform that you can switch to?

Regards

Hagen

Re: Piler 1.3.10 issues with imap importing (Lotus Domino Server) from GUI

[sj]

Hello Hugo,

I'm so sorry that the fix took quite some time to be made. Please
get this commit 
https://bitbucket.org/jsuto/piler/commits/8685574ede416cfce65fb8b1ebc950764778c898

and it solves this issue.

Janos


On 2021-01-27 16:35, Hugo Alvarez wrote:

Hi Janos,

First thanks for all.

Regarding the python version, sorry for the mistake, we are indeed
using python3 (3.6).

Regarding the problem with the import script, the process is actually
finished correctly, the issue is that the relevant fields in the
'import' database table does not seem to get updated.

Upon creation of the task through the gui, the first run of the
import.sh script will actually import all the emails, but only the
following fields get updated in the database:

status: 2 (from imapfetch.py and then from import.c)

total: abc (from imapfetch.py)
imported: xyz (from import.c)

The problem here is that when imapfech.py runs again (when cron reruns
import.sh), the executed query is:

SELECT id, server, username, password FROM import WHERE started=0

which in turn will return all the import jobs again.

Hope this makes the problem clear.

Thanks

Regards

 [1]
Libre de virus. www.avast.com [1]

El mar, 26 ene 2021 a las 20:34,  escribió:


Hello Hugo,

thank you for the pacthes. Have you tried to use imapfetch.py with
python3?
Also can you track what happens or at what point pilerimport stucks?

Anything
relevant in the mail logs?

I've just re-run the import test, and pilerimport has properly
updated
the
import table.

Janos

On 2021-01-25 13:27, Hugo Alvarez wrote:


We have installed Piler 1.3.10 and we cannot get imap importing

(Lotus

Domino server) from GUI. Here is what we have been able to

trackdown

so far:

1 - In read_folder_list we had to decode to utf-8  the case for

the

tupple type. A type missmatch error would occur otherwise:

elif isinstance(folder, type(())):
folder = re.sub(r'\{\d+\}$', '',
folder[0].decode('utf-8')) + folder[1].decode('utf-8')

2 - Domino's Imap server returns a backslash the field separator:

[b'(\\Noinferiors \\HasNoChildren) "" Inbox', b'(\\Noinferiors
\\HasNoChildren) "" Trash', b'(\\HasNoChildren) ""

Drafts',

b'(\\HasNoChildren) "" Sent']

Which does not match against the exiting regexp in

read_folder_list in

imapfetch.py. This is fixed adding the backslash to the regexp:  f

=

re.split(r' \"[\/\.\\]+\" ', folder)

3 - We are using python 2.7.5 under Centos 7.9.2009 which seens to
have a bug in imaplib's _checkquote(), which prevents folder names
with special characters from being correctly escaped making
imapfetch.py to fail. We fixed this by using a  sane _checkquote()
version from a previous imaplib. Maybe there is a cleaner solution

...


result.append(_checkquote(conn, f[1]))

4 - In process_folder, empty emails will fail when trying to write

to

disk. This is fixed by checking for a tupple:

for num in data[0].split():
rc, data = conn.fetch(num, '(RFC822)')
if opts['verbose']:
print(rc, num)
if isinstance(data[0], tuple):
opts['counter'] += 1
with open("{}.eml".format(opts['counter']), "wb")

as

f:f.write(data[0][1])

We could make it work with these changes, but pilerimport -T does

not

seem to mark the fields 'updated' or 'finished' in the import

table,

so subsequent executions of import.sh starts the whole process

again

(failing the import due to duplicates).




Links:
--
[1] 
https://www.avast.com/sig-email?utm_medium=emailutm_source=linkutm_campaign=sig-emailutm_content=webmail

Re: Piler auditor login error

[sj]

Hello,

On 2021-06-25 21:34, Marcelo Machado wrote:


I did a fresh installation of Mail Piler version 1.3.11 on Ubuntu
16.04 and when I log in with auditor user, the error message "HTTP
ERROR 500" appears.

The apache error log shows the error "PHP Parse error: syntax error,
unexpected '=' in /var/piler/www/model/search/search.php on line 734"


Though I use nginx and not apache, I can't reproduce the issue.
Anyway, try commenting out this line.

Janos

Re: Pilerexport attachment numbers plus advanced querying

[sj]

OK, this commit gives you the raw: label: 
https://bitbucket.org/jsuto/piler/commits/7b47ce3ca20f9f4ab3234afd6641fe364eaedbc6


Whatever you enter after 'raw:', it will be inside the MATCH('') clause.

Janos

On 2021-05-02 09:15, Ryan Blenis wrote:

Hi Janos,

On Sun, May 2, 2021 at 2:24 AM  wrote:


Hello Ryan,

On 2021-05-02 01:15, Ryan Blenis wrote:


1. Is there a way to have the pilerexport dry run output the

number of

attachments linked to the query?


Just to clarify. Let's say you have 10 matching emails with 15
attachments
total, and you want pilerexport to print 15?


Ideally, in the dry run, I'd like to have it print "attachment: [id]"
much like the emails print "id: [id]" so I can get a count of emails
and associated attachment numbers, which is often required in legal
queries as a the "number of items" found for certain search terms.
(I'm on a slightly older version that only prints "id: [id]" for each
email found, though I believe you added an email found count at the
end of the dry run in a newer version, ideally in that case there
would be a count of emails, and a count of attachments as well).
Thank you!


2. Would it be possible in the web UI to have an "expert"
setting/field where you can manually enter the full sphinx MATCH()
string for more precise results (while still utilizing the other
advanced search fields)? The sphinx query language allows quite a

bit

more flexibility and I'm finding myself using pilerexport's -w

flag to

get a read on match numbers versus querying in the web UI because

of

this limitation (even though you CAN overload the fields to an

extent)

It's possible with a little tweak, I'll add it soon, but it makes
sense
only for auditors. Otherwise it breaks security.


Yes, I completely agree. I'm usually in as an auditor so I almost
forgot about other roles, thank you!


2a. Overloading the web UI search fields with a more complex

query, or

even simply utilizing "quotes" in the field appears to break the
"sphinx" query viewing link. This appears to be just a simple need

for

some escaping on the front-end.


I've fixed the issue with this commit:


https://bitbucket.org/jsuto/piler/commits/dacb7c85aa7654c70f97c9bd4dc80a52913d842a

Awesome! Thank you!


Janos

Re: Pilerexport attachment numbers plus advanced querying

[sj]

Hello Ryan,

On 2021-05-02 01:15, Ryan Blenis wrote:


1. Is there a way to have the pilerexport dry run output the number of
attachments linked to the query?


Just to clarify. Let's say you have 10 matching emails with 15 
attachments

total, and you want pilerexport to print 15?


2. Would it be possible in the web UI to have an "expert"
setting/field where you can manually enter the full sphinx MATCH()
string for more precise results (while still utilizing the other
advanced search fields)? The sphinx query language allows quite a bit
more flexibility and I'm finding myself using pilerexport's -w flag to
get a read on match numbers versus querying in the web UI because of
this limitation (even though you CAN overload the fields to an extent)


It's possible with a little tweak, I'll add it soon, but it makes sense
only for auditors. Otherwise it breaks security.


2a. Overloading the web UI search fields with a more complex query, or
even simply utilizing "quotes" in the field appears to break the
"sphinx" query viewing link. This appears to be just a simple need for
some escaping on the front-end.


I've fixed the issue with this commit:
https://bitbucket.org/jsuto/piler/commits/dacb7c85aa7654c70f97c9bd4dc80a52913d842a

Janos

Re: AW: Restore to O365 fails

[sj]

Hello Martin,

On 2021-04-27 15:23, Martin Nadvornik wrote:


we also have the same or at least a similar problem. However not with
every mail but with most. We were so far also unable to find the root
cause for this and can't see any distinctive differences between mails
that fail to restore and those that don't. In our specific setup piler
is using a postfix server as smart host which then redirects the
messages to exchange online via a transport rule. EXO accepts the
restored messages our postfix sends but never delivers them to the
inbox.
Looking at the message trace in EXO shows that the message is beeing
recived but there are no detailed trace infos available.

Out first guess was also that EXO doesn't like recveiving messages
with the same message ID multiple times. We haven't tried
REWRITE_MESSAGE_ID yet but based on what Alex wrote this seems to not
solve the problem.


Even then it would worth to check. Based on your inputs it's a pretty
non-deterministic issue. If you have the time and the patience file a
support issue for O365, and let's see what they have to say about it.

Janos

Re: O365 - ADSync and groups

[sj]

On 2021-04-17 00:36, Alexander Noack wrote:

I created a group in Piler by the name of a Public Folder and assigned
the email addresses from the rcpt table for that Public Folder.

Now I am trying to add the Group to the user session via
CUSTOM_EMAIL_QUERY_FUNCTION but apparently the auth_data doesn't have
a group field... :(

I would like to avoid having to add all email addresses of the Public
Folders, is there any way to add a group to the session data?


What's the ultimate goal? Merely adding another group to the user or to
add the group address to the user? Assuming it's the latter, then you
may skip adding the group, simply just add the Public Folder email 
addresses

to the selected users.

Anyway, I may extend the auth_data array with the groups, if it's the
more elegant and effective solution to your problem.

Janos



Piler is setup to do Authentication against MS-ActiveDirectory. All
Mailboxes are hosted on O365, sent to Piler via a Journaling rule.
AD-Sync is setup to sync with Azure-Ad (O365).

Now when I login to Piler (Auto-Login via SSO) I don't see any
additional groups for my user. It looks like for Exchange
On-Premises this attribute would get populated by my
distribution-groups membership. Since Groups created in O365 are not
synced back, they don't exist in my AD.

I know, I can use
https://www.mailpiler.org/wiki/current:custom-authentication to
create group attributes based on arbitrary AD groups. I just would
like to know how other people are tackling that issue.

For Public-Folder email addresses I created Groups within Piler and
assigned users there.

Alex

Re: pilerexport w flag matching

[sj]

Hello Ryan,

try getting the master branch, do a complete build, however, don't 
install it.
Instead enter the src directory, use LD_LIBRARY_PATH=. to make sure 
./pilerexport

will use the recently built libpiler.so file, and try again. Also it's
worth to check what's going on 127.0.0.1:9306 with ngrep or similar 
tool.


Janos

On 2021-04-08 08:45, Ryan Blenis wrote:

Hi Janos,

Thanks as always for the reply. No worries on the -w replacing the
other switches, a documentation update will clear that up pretty
easily, and thank you, I'd love a total in the dry run!

As for "the export utility assumes searchd is listening on
127.0.0.1:9306 [1] That part didn't change from 1.3.8 to 1.3.11."

I looked back at all the commits and didn't see anything in that file
that could be causing it, but I'll investigate because now it's just
annoying me that it doesn't work. I saw that it was assuming searchd
was on 127.0.0.1:9306 [1], and it is on my system, however it still
fails with that error in the new build. I don't know if you have a
test case for pilerexport -w, but it should be repeatable. I don't see
much changing in the cfg.c either, but when I printf the cfg.mysqluser
and cfg.mysqlpwd on the old (1.3.8) version I get the correct info. If
I printf cfg.mysqluser and cfg.mysqlpwd on the new version I get "r"
and "" respectively. A quick strace of pilerexport shows it opening
the correct conf file (/usr/local/etc/piler/piler.conf) on both
versions. And specifying the config file with -c did not help either.
Not sure if you can reproduce this on your end or not.

The odd part is, cfg.mysqluser and cfg.mysqlpwd _IS_ utilized in the
"init_session_data(, );" line above the
"init_session_data(, );" and it passes the database "open"
test just fine there...

Thank you.

On Thu, Apr 8, 2021 at 12:13 AM  wrote:


Hello Ryan,

On 2021-04-08 01:36, Ryan Blenis wrote:


Thanks, that led me to what is causing the issue / confusion.

The -w switch is described as "Where condition to pass to sphinx,

eg.

"match('@subject: piler')"

Which led me to believe the MATCH string was all that was supposed

to

be there/replaced, however a quick look at the code shows that if

-w

is used, it REPLACES the ENTIRE where clause. This distinction

means

yes, perhaps the "eg." was not that prominent in the short --help
output.
I'll improve the docs on the website, it's lagging behind the actual

features,
and I'll add a clarification on it.


The simplest workaround to this for others would be to note that

-w

allows you to build your own query and negates the use of other
parameters. The ideal fix I think would be to still utilize the

other

parameters, but have -w content appended within the MATCH()

portion of

the query.


Such fix would only complicate things because you can define the
whole
query using -w including the time frame, recipients, etc. Again,
I'll
add a clarification to the docs.


Aside from that: I realize I'm behind on piler (1.3.8), and would

like

to update to get the latest pilerexport with zip capabilities, yet

I

see there is no upgrade information on
https://www.mailpiler.org/wiki/current:upgrade . What is the

process

to the latest (1.3.11)?


Well, simply compile the new stuff, and overwrite the binaries and
the
GUI files. The database schema hasn't changed from 1.3.8 to 1.3.11.
However, don't rush with that. The zip export feature has a poor
performance that needs a rework.


I'd also like to add a "--num-only" type flag to pilerexport to

see

the number of matches before exporting (would probably imply

dryrun).

We have something similar. When specifying -d (or --dry-run) it
prints
the matching serial ids, eg.

$ pilerexport -d -w "MATCH(' some query')"
id:318
id:375
id:518
id:656
id:660
id:688
id:733

I'll improve it to add "total:7" as the last line of the output, if
that's OK.


When trying to just compile the latest, I get the error "error:

cannot

connect to 127.0.0.1:9306 [1] [1]" so I'm not sure if that's an

issue

because not all the components are upgraded, or if I had a

different

configure flag/path configured during the original install.


The export utility assumes searchd is listening on 127.0.0.1:9306
[1]
That part didn't change from 1.3.8 to 1.3.11.

Janos



Links:
--
[1] http://127.0.0.1:9306

Re: pilerexport w flag matching

[sj]

Hello Ryan,

On 2021-04-08 01:36, Ryan Blenis wrote:


Thanks, that led me to what is causing the issue / confusion.

The -w switch is described as "Where condition to pass to sphinx, eg.
"match('@subject: piler')"

Which led me to believe the MATCH string was all that was supposed to
be there/replaced, however a quick look at the code shows that if -w
is used, it REPLACES the ENTIRE where clause. This distinction means


yes, perhaps the "eg." was not that prominent in the short --help 
output.
I'll improve the docs on the website, it's lagging behind the actual 
features,

and I'll add a clarification on it.


The simplest workaround to this for others would be to note that -w
allows you to build your own query and negates the use of other
parameters. The ideal fix I think would be to still utilize the other
parameters, but have -w content appended within the MATCH() portion of
the query.


Such fix would only complicate things because you can define the whole
query using -w including the time frame, recipients, etc. Again, I'll
add a clarification to the docs.


Aside from that: I realize I'm behind on piler (1.3.8), and would like
to update to get the latest pilerexport with zip capabilities, yet I
see there is no upgrade information on
https://www.mailpiler.org/wiki/current:upgrade . What is the process
to the latest (1.3.11)?


Well, simply compile the new stuff, and overwrite the binaries and the
GUI files. The database schema hasn't changed from 1.3.8 to 1.3.11.
However, don't rush with that. The zip export feature has a poor
performance that needs a rework.


I'd also like to add a "--num-only" type flag to pilerexport to see
the number of matches before exporting (would probably imply dryrun).


We have something similar. When specifying -d (or --dry-run) it prints
the matching serial ids, eg.

$ pilerexport -d -w "MATCH(' some query')"
id:318
id:375
id:518
id:656
id:660
id:688
id:733

I'll improve it to add "total:7" as the last line of the output, if
that's OK.


When trying to just compile the latest, I get the error "error: cannot
connect to 127.0.0.1:9306 [1]" so I'm not sure if that's an issue
because not all the components are upgraded, or if I had a different
configure flag/path configured during the original install.


The export utility assumes searchd is listening on 127.0.0.1:9306
That part didn't change from 1.3.8 to 1.3.11.

Janos

Re: pilerexport w flag matching

[sj]

Hello Ryan,

please apply this patch to pilerexport.c, and recompile it.
https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d

It syslogs the sphinx query. Then try the following. Enter the search 
query

on the gui, and record the sphinx query syslogged. Then re-run the
pilerexport command, and record the new sphinx query, and compare it
with the previous value.

Verify that even the single-quotes and double quotes are the same in 
both queries.


Janos SUTO


On 2021-04-07 18:18, Ryan Blenis wrote:

Hi Janos,

I have to export potentially a ton of emails and was looking to use
pilerexport versus multiple batches of GUI searches. I saw the -w flag
and thought "great, I can use this" but it doesn't seem to respond
appropriately for my test case. I have 2 emails that match the
following (generalized terms used vs actual), limiting with -m 3 for
testing purposes (I should only get 2 back).

pilerexport -a 2010.10.01 -b 2021.04.06 -r "j...@domain.com" -m 3 -w
'MATCH('"'"'searchterm NEAR/25 (MNF|(search term)|term|(test search
term)|termin*)'"'"')'

Now, that match is just the bash string escaped version of:
MATCH('searchterm NEAR/25 (MNF|(search term)|term|(test search
term)|termin*)')
(That's just a fancy sphinx query for "searchterm" within 25 words of
MNF OR "search term" OR "term" OR "test search term" or "termin*" for
those unfamiliar with sphinx.)

Which, when overloading the Advanced Search for the "body" field in
the GUI with:
searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*)

Seems to work just fine and as expected, however, in pilerexport with
the aforementioned command I get tons of unrelated emails (not even
scoped to the appropriate j...@domain.com recipient). Is using a MATCH
term like this with -w possible, or am I looking to do too much here?

Note that I saw you added the -o parameter in the source so I may be a
version or 2 back (utility doesn't seem to have a -v or --version
output), and my version doesn't appear to have that, so I don't really
have any great diagnostic/output information to go off of other than
the above description.

Thank you in advance as always for any insight you can give!

Re: Problem after Ubuntu Upgrade from 16.04 to 18.04

[sj]

Hello Frank,

in openssl 1.1.x the output of the openssl command has changed. Assuming
your mentioned piler installation is older than 2020-10-17, then I 
suggest
to try updating system/helper/TrustedTimestamps.php from the master 
branch,

and let's see how it goes.

Janos

On 2021-04-06 16:50, Frank Schmitz wrote:

Hello all,

I had to perform an Ubuntu Upgrade to 18.04 on a server running an
older version of piler.

The update ran without any problems, but the piler Web-GUI seems to be
"broken".

I get the following error in the apache log (pseudo anonymized):

[Tue Apr 06 16:35:53.008548 2021] [php7:error] [pid 14368] [client
xxx:64562] PHP Fatal error:  Uncaught Exception: Systemcommand failed:
Using configuration from /usr/lib/ssl/openssl.cnf, Verification: OK in
/var/www/xxx/htdocs-ssl/system/helper/TrustedTimestamps.php:186\nStack
trace:\n#0 /var/www/xxx/htdocs-ssl/model/search/message.php(512):
TrustedTimestamps::validate('1c79bef0265fxxx...',
'MIIVSzADAgEAxxx...', '1617716xxx', '/var/piler/free...')\n#1
/var/www/xxx/htdocs-ssl/model/search/message.php(51):
ModelSearchMessage->check_rfc3161_timestamp_for_id('2519')\n#2
/var/www/xxx.com/htdocs-ssl/model/search/message.php(228):
ModelSearchMessage->verify_message('4000606xxx8...', 'Return-Path:
extract_message('4000606cxxx...', '')\n#4
/var/www/xxx/htdocs-ssl/system/front.php(36):
ControllerMessageView->index()\n#5
/var/www/xxx/htdocs-ssl/system/front.php(14): Fro in
/var/www/xxx/htdocs-ssl/system/helper/TrustedTimestamps.php on line
186

I see that TrustedTimestamps.php throws an exception, but I don't
understand why...
It says "Verification: OK" which should indicate that everything has
gone well, right?

openssl is installed (1.1.1), the cron job signing messages using
TrustedTimestamps.php is also running fine.

But using the gui throws an exception...

Help anyone?

Kind regards
Frank Schmitz

Re: watch_sphinx_main_index.sh - File is missing?

[sj]

Hello BKH,

you are right. The postinstall created cron entries indeed have
watch_sphinx_main_index.sh, however such script doesn't exist.

The original purpose of the script would be to display a warning
on the health page when the main1.* files have been grown pretty
large and it's worth to open main2.

I'll create this script, until then I suggest you to comment out
this cron entry.

Thank you for noticing this issue ;-)

Janos

On 2021-03-08 01:35, BKH wrote:

Hello Janos,

i did a fresh install of Piler 1.3.11 an got some errors regarding
cronjob  "3 * * * *
/usr/local/libexec/piler/watch_sphinx_main_index.sh“
After looking into "/usr/local/libexec/piler/" i found out that
„watch_sphinx_main_index.sh“ is missing.

Just to be sure i did a fesh install of piler 1.3.10 and noticed that
the same file ist missing too.
Even the developer version shows this behavior.

Is this the expected way when there is a entry in cronjob which calls 
this file?


BKH

Renaming the group table

[sj]

Dear piler users,

I'd like to get your help with 
https://bitbucket.org/jsuto/piler/issues/1058/database-table-name-group-reserved-keyword


The issue is about finding a better name for the group table.

The problem is that "group" is a reserved keyword in sql, so
you need to backtick it in a query like select * from `group`;

The reason that I picked this reserved word back then is because
I didn't find a better word for describing a group of users than
'group'.

I've just looked up https://www.thesaurus.com/browse/group?s=t
and search for synonyms, but I'm stuck. I don't want to come up
with a silly word instead of 'group'.

So I'd like you to vote to either keep the table name as group
and use `group` in your queries or recommend a better word. In
the latter case we need to rename the group table to the new name
which is a trivial task, though still some manual intervention
at an upgrade to the upcoming 1.3.12 (I haven't started working
on it yet).

Janos

piler 1.3.11 has been released

[sj]

Dear piler users,

I've just released 1.3.11, a bugfix release addressing the crash of 
piler-smtp daemon.
It occurs under certain conditions, usually when piler runs in a virtual 
machine, and
it's suspended by a backup application while processing an email. When 
the VM is resumed
then piler-smtp finds that it's a timeout session, then it manages to 
finish the smtp
transaction, writes the email to disk, then it gets a SIGSEGV signal, 
and it terminates.


I've refactored the timeout checking code in 1.3.11 to fix this problem. 
Also I've changed
the LDAP parameters in the piler GUI. By now the port settings as a 
dedicated parameter
has been deprecated. You should specify the ldap uri in the 
ldap://your.ldap.host:port

format.

Please find the downloadable artifacts at 
https://www.mailpiler.org/wiki/download
including the source tarball, deb and rpm packages as well as the docker 
image.


Janos

Re: AW: Piler 1.35 on Ubuntu 18.04 - piler-smtp SIGSEGV

[sj]

On 2021-02-22 14:19, Wagner, Patrick wrote:

We can probably get rid of the tempory debug output code, can't we?


Yes. Either remove them for yourself or get the master branch,
and recompile, then overwrite the piler-smtp binary.

Janos


On 22.02.2021 13:17, s...@acts.hu wrote:



Hello,

thank you for both of your feedbacks. As much as I dislike systemd,
it allows you to restart piler-smtp in case it crashes. But hopefully
it won't happen in the future after this fix.

Also I've merged it to the master branch, and I'll release a bugfix
version 1.3.11, and I'll make a deb package.

Janos


On 2021-02-22 12:11, claudio.schnelldasi...@manrolandgoss.com wrote:

Janos, Patrick,

thanks for this. I also run piler on Ubuntu 18.04 and ran into the 
same issue.

My 'solution' was to start piler-smtp automatically via monitoring if
it goes down :-(.

Janos, I wonder if you will provide updated .deb packages for 
dpkg-installs?


Thanks and regards,
Claudio

-Ursprüngliche Nachricht-
Von: Wagner, Patrick 
Gesendet: Montag, 22. Februar 2021 12:01
An: Piler User 
Betreff: Re: Piler 1.35 on Ubuntu 18.04 - piler-smtp SIGSEGV



Thanks Janos,

I'd updated and replaced the piler-smtp binary on 2021-02-16, and
while I can see that my monitoring still shows SMTP timeouts on the
piler port sometimes between 02:30 and 02:40, no further outright
crashes / SIGSEGV occur, so archiving continues without any need to
restart the process manually.


- Patrick

On 15.02.2021 21:49, s...@acts.hu wrote:



Hello Patrick,

thank you for your feedback. I'd like you to update the smtpsegv
branch, and recompile piler-smtp. I've refactored the timeout 
checks,

and pushed the commit to the branch on bitbucket.

Janos


On 2021-02-15 14:34, Wagner, Patrick wrote:

Hello Janos,


happened Sa->So this time.


Logs (piler-smtp as compiled out of smtpsegv.tar.gz source code) 
see

below, SIGSEGV follows right after free(session) done.

It is also interesting to see that the piler daemon finishes
processing a message 2 whole minutes after the smtp process has
already segfaulted and and stopped supplying any new messages.


Thanks, Patrick



Feb 14 02:29:59 hostname piler-smtp[19066]: connected from
192.168.165.154:9501 on fd=6 (active connections: 1) Feb 14 
02:29:59

hostname piler-smtp[19066]: received:
Q056C70SAJJT9OFH, from=, size=4944, client=192.168.165.154, fd=6,
fsync=1831
Feb 14 02:29:59 hostname piler-smtp[19066]: disconnected from
192.168.165.154 on fd=6, slot=0, reason=done (0 active connections)
Feb 14 02:29:59 hostname piler-smtp[19066]: free_smtp_session() Feb
14 02:29:59 hostname piler-smtp[19066]: SSL_shutdown() Feb 14
02:29:59 hostname piler-smtp[19066]: SSL_free() Feb 14 02:29:59
hostname piler-smtp[19066]: SSL_CTX_free Feb 14 02:29:59 hostname
piler-smtp[19066]: freeing session Feb 14 02:29:59 hostname
piler-smtp[19066]: free(session) done Feb 14 02:30:00 hostname
piler[19070]: 1/Q056C70SAJJT9OFH:
400060287d2205ab9b5400229f77e67a, size=4944/1928, 
attachments=0,

reference=, message-id=<4DdV5z0DlSz 5vfP@mailrelay.intern>,
retention=7350, folder=0, delay=0.0425, status=stored Feb 14 
02:30:01

hostname CRON[7835]: (piler) CMD (/usr/bin/find /var/www/piler/tmp
-type f -name i.\* -exec rm -f {} \;) Feb 14 02:30:01 hostname
CRON[7836]: (piler) CMD (/usr/bin/find /var/piler/error -type f|wc 
-l

> /var/piler/stat/error) Feb 14 02:30:01 hostname CRON[7842]: (piler)
CMD (/usr/bin/indexer --quiet note1 --rotate) Feb 14 02:30:01
hostname CRON[7837]: (piler) CMD (/usr/bin/indexer --quiet tag1
--rotate) Feb 14 02:30:01 hostname CRON[7838]: (piler) CMD
(/usr/local/libexec/piler/indexer.main.sh)
Feb 14 02:30:01 hostname CRON[7833]: (CRON) info (No MTA installed,
discarding output) Feb 14 02:30:01 hostname CRON[7832]: (CRON) info
(No MTA installed, discarding output) Feb 14 02:30:01 hostname 
piler:

INDEXER INFO: merging to main started Feb 14 02:30:40 hostname
piler-smtp[19066]: connected from
192.168.165.252:37934 on fd=6 (active connections: 1) Feb 14 
02:30:41

hostname piler-smtp[19066]: connected from
192.168.165.252:37936 on fd=9 (active connections: 2) Feb 14 
02:30:41

hostname piler-smtp[19066]: disconnected from
192.168.165.252 on fd=6, slot=0, reason=done (1 active connections)
Feb 14 02:30:41 hostname piler-smtp[19066]: free_smtp_session() Feb
14 02:30:41 hostname piler-smtp[19066]: SSL_shutdown() Feb 14
02:30:41 hostname piler-smtp[19066]: SSL_free() Feb 14 02:30:41
hostname piler-smtp[19066]: SSL_CTX_free Feb 14 02:30:41 hostname
piler-smtp[19066]: freeing session Feb 14 02:30:41 hostname
piler-smtp[19066]: free(session) done Feb 14 02:30:41 hostname
piler-smtp[19066]: disconnected from
192.168.165.252 on fd=9, slot=1, reason=done (0 active connections)
Feb 14 02:30:41 hostname piler-smtp[19066]: free_smtp_session() Feb
14 02:30:41 hostname piler-smtp[19066]: SSL_shutdown() Feb 14
02:30:41 hostname piler-smtp[19066]: SSL_free() Feb 14 02:30:41
hostname piler-smtp[19066]: SSL_CTX_free Feb 14 02:30:41 

Re: AW: Piler 1.35 on Ubuntu 18.04 - piler-smtp SIGSEGV

[sj]

Hello,

thank you for both of your feedbacks. As much as I dislike systemd,
it allows you to restart piler-smtp in case it crashes. But hopefully
it won't happen in the future after this fix.

Also I've merged it to the master branch, and I'll release a bugfix
version 1.3.11, and I'll make a deb package.

Janos


On 2021-02-22 12:11, claudio.schnelldasi...@manrolandgoss.com wrote:

Janos, Patrick,

thanks for this. I also run piler on Ubuntu 18.04 and ran into the same 
issue.

My 'solution' was to start piler-smtp automatically via monitoring if
it goes down :-(.

Janos, I wonder if you will provide updated .deb packages for 
dpkg-installs?


Thanks and regards,
Claudio

-Ursprüngliche Nachricht-
Von: Wagner, Patrick 
Gesendet: Montag, 22. Februar 2021 12:01
An: Piler User 
Betreff: Re: Piler 1.35 on Ubuntu 18.04 - piler-smtp SIGSEGV



Thanks Janos,

I'd updated and replaced the piler-smtp binary on 2021-02-16, and
while I can see that my monitoring still shows SMTP timeouts on the
piler port sometimes between 02:30 and 02:40, no further outright
crashes / SIGSEGV occur, so archiving continues without any need to
restart the process manually.


- Patrick

On 15.02.2021 21:49, s...@acts.hu wrote:



Hello Patrick,

thank you for your feedback. I'd like you to update the smtpsegv
branch, and recompile piler-smtp. I've refactored the timeout checks,
and pushed the commit to the branch on bitbucket.

Janos


On 2021-02-15 14:34, Wagner, Patrick wrote:

Hello Janos,


happened Sa->So this time.


Logs (piler-smtp as compiled out of smtpsegv.tar.gz source code) see
below, SIGSEGV follows right after free(session) done.

It is also interesting to see that the piler daemon finishes
processing a message 2 whole minutes after the smtp process has
already segfaulted and and stopped supplying any new messages.


Thanks, Patrick



Feb 14 02:29:59 hostname piler-smtp[19066]: connected from
192.168.165.154:9501 on fd=6 (active connections: 1) Feb 14 02:29:59
hostname piler-smtp[19066]: received:
Q056C70SAJJT9OFH, from=, size=4944, client=192.168.165.154, fd=6,
fsync=1831
Feb 14 02:29:59 hostname piler-smtp[19066]: disconnected from
192.168.165.154 on fd=6, slot=0, reason=done (0 active connections)
Feb 14 02:29:59 hostname piler-smtp[19066]: free_smtp_session() Feb
14 02:29:59 hostname piler-smtp[19066]: SSL_shutdown() Feb 14
02:29:59 hostname piler-smtp[19066]: SSL_free() Feb 14 02:29:59
hostname piler-smtp[19066]: SSL_CTX_free Feb 14 02:29:59 hostname
piler-smtp[19066]: freeing session Feb 14 02:29:59 hostname
piler-smtp[19066]: free(session) done Feb 14 02:30:00 hostname
piler[19070]: 1/Q056C70SAJJT9OFH:
400060287d2205ab9b5400229f77e67a, size=4944/1928, attachments=0,
reference=, message-id=<4DdV5z0DlSz 5vfP@mailrelay.intern>,
retention=7350, folder=0, delay=0.0425, status=stored Feb 14 02:30:01
hostname CRON[7835]: (piler) CMD (/usr/bin/find /var/www/piler/tmp
-type f -name i.\* -exec rm -f {} \;) Feb 14 02:30:01 hostname
CRON[7836]: (piler) CMD (/usr/bin/find /var/piler/error -type f|wc -l
> /var/piler/stat/error) Feb 14 02:30:01 hostname CRON[7842]: (piler)
CMD (/usr/bin/indexer --quiet note1 --rotate) Feb 14 02:30:01
hostname CRON[7837]: (piler) CMD (/usr/bin/indexer --quiet tag1
--rotate) Feb 14 02:30:01 hostname CRON[7838]: (piler) CMD
(/usr/local/libexec/piler/indexer.main.sh)
Feb 14 02:30:01 hostname CRON[7833]: (CRON) info (No MTA installed,
discarding output) Feb 14 02:30:01 hostname CRON[7832]: (CRON) info
(No MTA installed, discarding output) Feb 14 02:30:01 hostname piler:
INDEXER INFO: merging to main started Feb 14 02:30:40 hostname
piler-smtp[19066]: connected from
192.168.165.252:37934 on fd=6 (active connections: 1) Feb 14 02:30:41
hostname piler-smtp[19066]: connected from
192.168.165.252:37936 on fd=9 (active connections: 2) Feb 14 02:30:41
hostname piler-smtp[19066]: disconnected from
192.168.165.252 on fd=6, slot=0, reason=done (1 active connections)
Feb 14 02:30:41 hostname piler-smtp[19066]: free_smtp_session() Feb
14 02:30:41 hostname piler-smtp[19066]: SSL_shutdown() Feb 14
02:30:41 hostname piler-smtp[19066]: SSL_free() Feb 14 02:30:41
hostname piler-smtp[19066]: SSL_CTX_free Feb 14 02:30:41 hostname
piler-smtp[19066]: freeing session Feb 14 02:30:41 hostname
piler-smtp[19066]: free(session) done Feb 14 02:30:41 hostname
piler-smtp[19066]: disconnected from
192.168.165.252 on fd=9, slot=1, reason=done (0 active connections)
Feb 14 02:30:41 hostname piler-smtp[19066]: free_smtp_session() Feb
14 02:30:41 hostname piler-smtp[19066]: SSL_shutdown() Feb 14
02:30:41 hostname piler-smtp[19066]: SSL_free() Feb 14 02:30:41
hostname piler-smtp[19066]: SSL_CTX_free Feb 14 02:30:41 hostname
piler-smtp[19066]: freeing session Feb 14 02:30:41 hostname
piler-smtp[19066]: free(session) done Feb 14 02:31:44 hostname
piler-smtp[19066]: connected from
192.168.165.252:40584 on fd=6 (active connections: 1) Feb 14 02:31:53
hostname piler-smtp[19066]: connected from
192.168.165.252:40586 on 

Re: Piler 1.35 on Ubuntu 18.04 - piler-smtp SIGSEGV

[sj]

Hello Patrick,

thank you for your feedback. I'd like you to update the smtpsegv branch,
and recompile piler-smtp. I've refactored the timeout checks, and pushed
the commit to the branch on bitbucket.

Janos


On 2021-02-15 14:34, Wagner, Patrick wrote:

Hello Janos,


happened Sa->So this time.


Logs (piler-smtp as compiled out of smtpsegv.tar.gz source code) see
below, SIGSEGV follows right after free(session) done.

It is also interesting to see that the piler daemon finishes
processing a message 2 whole minutes after the smtp process has
already segfaulted and and stopped supplying any new messages.


Thanks, Patrick



Feb 14 02:29:59 hostname piler-smtp[19066]: connected from
192.168.165.154:9501 on fd=6 (active connections: 1)
Feb 14 02:29:59 hostname piler-smtp[19066]: received:
Q056C70SAJJT9OFH, from=, size=4944, client=192.168.165.154, fd=6,
fsync=1831
Feb 14 02:29:59 hostname piler-smtp[19066]: disconnected from
192.168.165.154 on fd=6, slot=0, reason=done (0 active connections)
Feb 14 02:29:59 hostname piler-smtp[19066]: free_smtp_session()
Feb 14 02:29:59 hostname piler-smtp[19066]: SSL_shutdown()
Feb 14 02:29:59 hostname piler-smtp[19066]: SSL_free()
Feb 14 02:29:59 hostname piler-smtp[19066]: SSL_CTX_free
Feb 14 02:29:59 hostname piler-smtp[19066]: freeing session
Feb 14 02:29:59 hostname piler-smtp[19066]: free(session) done
Feb 14 02:30:00 hostname piler[19070]: 1/Q056C70SAJJT9OFH:
400060287d2205ab9b5400229f77e67a, size=4944/1928, attachments=0,
reference=, message-id=<4DdV5z0DlSz
5vfP@mailrelay.intern>, retention=7350, folder=0, delay=0.0425, 
status=stored

Feb 14 02:30:01 hostname CRON[7835]: (piler) CMD (/usr/bin/find
/var/www/piler/tmp -type f -name i.\* -exec rm -f {} \;)
Feb 14 02:30:01 hostname CRON[7836]: (piler) CMD (/usr/bin/find
/var/piler/error -type f|wc -l > /var/piler/stat/error)
Feb 14 02:30:01 hostname CRON[7842]: (piler) CMD (/usr/bin/indexer
--quiet note1 --rotate)
Feb 14 02:30:01 hostname CRON[7837]: (piler) CMD (/usr/bin/indexer
--quiet tag1 --rotate)
Feb 14 02:30:01 hostname CRON[7838]: (piler) CMD
(/usr/local/libexec/piler/indexer.main.sh)
Feb 14 02:30:01 hostname CRON[7833]: (CRON) info (No MTA installed,
discarding output)
Feb 14 02:30:01 hostname CRON[7832]: (CRON) info (No MTA installed,
discarding output)
Feb 14 02:30:01 hostname piler: INDEXER INFO: merging to main started
Feb 14 02:30:40 hostname piler-smtp[19066]: connected from
192.168.165.252:37934 on fd=6 (active connections: 1)
Feb 14 02:30:41 hostname piler-smtp[19066]: connected from
192.168.165.252:37936 on fd=9 (active connections: 2)
Feb 14 02:30:41 hostname piler-smtp[19066]: disconnected from
192.168.165.252 on fd=6, slot=0, reason=done (1 active connections)
Feb 14 02:30:41 hostname piler-smtp[19066]: free_smtp_session()
Feb 14 02:30:41 hostname piler-smtp[19066]: SSL_shutdown()
Feb 14 02:30:41 hostname piler-smtp[19066]: SSL_free()
Feb 14 02:30:41 hostname piler-smtp[19066]: SSL_CTX_free
Feb 14 02:30:41 hostname piler-smtp[19066]: freeing session
Feb 14 02:30:41 hostname piler-smtp[19066]: free(session) done
Feb 14 02:30:41 hostname piler-smtp[19066]: disconnected from
192.168.165.252 on fd=9, slot=1, reason=done (0 active connections)
Feb 14 02:30:41 hostname piler-smtp[19066]: free_smtp_session()
Feb 14 02:30:41 hostname piler-smtp[19066]: SSL_shutdown()
Feb 14 02:30:41 hostname piler-smtp[19066]: SSL_free()
Feb 14 02:30:41 hostname piler-smtp[19066]: SSL_CTX_free
Feb 14 02:30:41 hostname piler-smtp[19066]: freeing session
Feb 14 02:30:41 hostname piler-smtp[19066]: free(session) done
Feb 14 02:31:44 hostname piler-smtp[19066]: connected from
192.168.165.252:40584 on fd=6 (active connections: 1)
Feb 14 02:31:53 hostname piler-smtp[19066]: connected from
192.168.165.252:40586 on fd=9 (active connections: 2)
Feb 14 02:31:53 hostname piler-smtp[19066]: disconnected from
192.168.165.252 on fd=6, slot=0, reason=done (1 active connections)
Feb 14 02:31:53 hostname piler-smtp[19066]: free_smtp_session()
Feb 14 02:31:53 hostname piler-smtp[19066]: SSL_shutdown()
Feb 14 02:31:53 hostname piler-smtp[19066]: SSL_free()
Feb 14 02:31:53 hostname piler-smtp[19066]: SSL_CTX_free
Feb 14 02:31:53 hostname piler-smtp[19066]: freeing session
Feb 14 02:31:53 hostname piler-smtp[19066]: free(session) done
Feb 14 02:31:53 hostname piler-smtp[19066]: disconnected from
192.168.165.252 on fd=9, slot=1, reason=done (0 active connections)
Feb 14 02:31:53 hostname piler-smtp[19066]: free_smtp_session()
Feb 14 02:31:53 hostname piler-smtp[19066]: SSL_shutdown()
Feb 14 02:31:53 hostname piler-smtp[19066]: SSL_free()
Feb 14 02:31:53 hostname piler-smtp[19066]: SSL_CTX_free
Feb 14 02:31:53 hostname piler-smtp[19066]: freeing session
Feb 14 02:31:53 hostname piler-smtp[19066]: free(session) done
Feb 14 02:32:40 hostname piler-smtp[19066]: connected from
192.168.165.252:43330 on fd=6 (active connections: 1)
Feb 14 02:32:40 hostname piler-smtp[19066]: connected from
192.168.165.252:43334 on fd=9 (active connections: 

Re: Piler 1.35 on Ubuntu 18.04 - piler-smtp SIGSEGV

[sj]

Hello Patrick,

then we need to dig deeper to understand what's going on. I created
a new branch to have some extra debugging. Please check out this commit:
https://bitbucket.org/jsuto/piler/commits/ac434e73ef86c031ee451284a00fc7062c579f9c

Perhaps it's easier to get 
https://bitbucket.org/jsuto/piler/get/smtpsegv.tar.gz

compile it, and overwrite only the piler-smtp binary.

Janos

On 2021-02-10 17:41, Wagner, Patrick wrote:

Hello Janos,


happened again, the same way, with the recompiled piler-smtp binary
(git master).

~# piler-smtp -V
1.3.10 build 998

Feb 10 02:30:01 hostname CRON[8429]: (piler) CMD (/usr/bin/indexer
--quiet note1 --rotate)
Feb 10 02:30:01 hostname CRON[8427]: (piler) CMD (/usr/bin/find
/var/piler/error -type f|wc -l > /var/piler/stat/error)
Feb 10 02:30:01 hostname CRON[8428]: (piler) CMD (/usr/bin/find
/var/www/piler/tmp -type f -name i.\* -exec rm -f {} \;)
Feb 10 02:30:01 hostname CRON[8431]: (piler) CMD
(/usr/local/libexec/piler/indexer.main.sh)
Feb 10 02:30:01 hostname CRON[8430]: (piler) CMD (/usr/bin/indexer
--quiet tag1 --rotate)
Feb 10 02:30:01 hostname CRON[8424]: (CRON) info (No MTA installed,
discarding output)
Feb 10 02:30:01 hostname CRON[8425]: (CRON) info (No MTA installed,
discarding output)
Feb 10 02:30:01 hostname piler: INDEXER INFO: merging to main started
Feb 10 02:30:17 hostname piler-smtp[27596]: connected from
192.168.165.252:57902 on fd=6 (active connections: 1)
Feb 10 02:30:17 hostname piler-smtp[27596]: connected from
192.168.165.252:57908 on fd=9 (active connections: 2)
Feb 10 02:30:17 hostname piler-smtp[27596]: disconnected from
192.168.165.252 on fd=6, slot=0 (1 active connections)
Feb 10 02:30:17 hostname piler-smtp[27596]: disconnected from
192.168.165.252 on fd=9, slot=1 (0 active connections)
Feb 10 02:30:31 hostname piler-smtp[27596]: connected from
192.168.165.154:32120 on fd=6 (active connections: 1)
Feb 10 02:31:47 hostname piler-smtp[27596]: received:
ZQAHOOJ8RXM5NCHD, from=, size=4945, client=192.168.165.154, fd=6,
fsync=75881359
Feb 10 02:31:47 hostname piler-smtp[27596]: client 192.168.165.154
timeout, lasttime: 1612920631
Feb 10 02:31:47 hostname piler-smtp[27596]: disconnected from
192.168.165.154 on fd=6, slot=0 (0 active connections)
Feb 10 02:31:47 hostname piler-smtp[27596]: got signal: 11, 
Segmentation fault
Feb 10 02:31:47 hostname piler-smtp[27596]: piler-smtp has been 
terminated

Feb 10 02:31:48 hostname piler[27599]: 0/ZQAHOOJ8RXM5NCHD:
40006023378d2c9c99d400a51d237bc0, size=4945/1936, attachments=0,
reference=, message-id=<4Db2JR32vQz5vS9@mailrelay.intern>,
retention=7350, folder=0, delay=0.9705, status=stored

There were no changes to backup schedules recently, and no scheduled
job or I/O heavy maintenance should happen, especially not at this
point in time. Still, point taken, we will check whether there could
be anything, but why would only 1-2 VMs out of hundreds be affected?

In any case, SIGSEGV still happens and my piler-smtp process is dead.

- Patrick

On 09.02.2021 09:04, s...@acts.hu wrote:



Hello Patrick,

I've been reported such issues when a backup system paused or froze
the virtual machine running piler. I suspect that it might be the
case for you based on the time and days you mentioned.

The problem in such cases is that piler-smtp is in the middle of
processing an email, then the system is frozen for a few minutes
enough to pass the smtp timeout limit (60 sec), and then when it's
unfrozen then piler-smtp notices the timeout situation, then tears
down the connection. However, at the end of the session it tries
to tear down the already terminated and freed connection data.

So, let's try the following: get the master branch, recompile piler,
however replace the piler-smtp binary only, nothing else. Then let's
see if the problem is solved.

Janos

On 2021-02-09 07:56, Wagner, Patrick wrote:

Hello everyone,

a few weeks ago 2 of our Piler instances began segfaulting - more
precisely, several times per month (but not every day night) at about
02:30 am local time the MS Exchange server that has the Piler 
instance

setup as journaling target connects to piler-smtp, upon which
piler-smtp segfaults after trying to store an email that clearly
doesn't get parsed correctly by piler-smtp as it is represented as
gibberish (and gibberish cannot be renamed...)

See below for a few of the latest examples. Interestingly, the 
SIGSEGV

did not happen Sunday and Monday night, meaning the day following a
non-weekday? Not going on long enough that I could call it a pattern,
but in any case it would be nice if piler-smtp didn't segfault in the
first place.

What can I do to supply more useful information to get this fixed? Or
has there been meaningful work on piler-smtp in the meantime that 
this

could / would've already been fixed?


Thanks, Patrick


Feb  4 02:30:01 hostname CRON[24428]: (piler) CMD (/usr/bin/find
/var/www/piler/tmp -type f -name i.\* -exec rm -f {} \;)
Feb  4 02:30:01 hostname CRON[24430]: (piler) 

Re: Piler 1.35 on Ubuntu 18.04 - piler-smtp SIGSEGV

[sj]

Hello Patrick,

I've been reported such issues when a backup system paused or froze
the virtual machine running piler. I suspect that it might be the
case for you based on the time and days you mentioned.

The problem in such cases is that piler-smtp is in the middle of
processing an email, then the system is frozen for a few minutes
enough to pass the smtp timeout limit (60 sec), and then when it's
unfrozen then piler-smtp notices the timeout situation, then tears
down the connection. However, at the end of the session it tries
to tear down the already terminated and freed connection data.

So, let's try the following: get the master branch, recompile piler,
however replace the piler-smtp binary only, nothing else. Then let's
see if the problem is solved.

Janos

On 2021-02-09 07:56, Wagner, Patrick wrote:

Hello everyone,

a few weeks ago 2 of our Piler instances began segfaulting - more
precisely, several times per month (but not every day night) at about
02:30 am local time the MS Exchange server that has the Piler instance
setup as journaling target connects to piler-smtp, upon which
piler-smtp segfaults after trying to store an email that clearly
doesn't get parsed correctly by piler-smtp as it is represented as
gibberish (and gibberish cannot be renamed...)

See below for a few of the latest examples. Interestingly, the SIGSEGV
did not happen Sunday and Monday night, meaning the day following a
non-weekday? Not going on long enough that I could call it a pattern,
but in any case it would be nice if piler-smtp didn't segfault in the
first place.

What can I do to supply more useful information to get this fixed? Or
has there been meaningful work on piler-smtp in the meantime that this
could / would've already been fixed?


Thanks, Patrick


Feb  4 02:30:01 hostname CRON[24428]: (piler) CMD (/usr/bin/find
/var/www/piler/tmp -type f -name i.\* -exec rm -f {} \;)
Feb  4 02:30:01 hostname CRON[24430]: (piler) CMD (/usr/bin/find
/var/piler/error -type f|wc -l > /var/piler/stat/error)
Feb  4 02:30:01 hostname CRON[24429]: (piler) CMD
(/usr/local/libexec/piler/indexer.main.sh)
Feb  4 02:30:01 hostname CRON[24431]: (piler) CMD (/usr/bin/indexer
--quiet note1 --rotate)
Feb  4 02:30:01 hostname CRON[24436]: (piler) CMD (/usr/bin/indexer
--quiet tag1 --rotate)
Feb  4 02:30:01 hostname piler: INDEXER INFO: merging to main started
Feb  4 02:30:01 hostname CRON[24426]: (CRON) info (No MTA installed,
discarding output)
Feb  4 02:30:01 hostname CRON[24425]: (CRON) info (No MTA installed,
discarding output)
Feb  4 02:30:20 hostname piler-smtp[1144]: connected from
192.168.165.252:58292 on fd=6 (active connections: 1)
Feb  4 02:30:20 hostname piler-smtp[1144]: connected from
192.168.165.252:58294 on fd=9 (active connections: 2)
Feb  4 02:30:20 hostname piler-smtp[1144]: disconnected from
192.168.165.252 on fd=6 (1 active connections)
Feb  4 02:30:20 hostname piler-smtp[1144]: disconnected from
192.168.165.252 on fd=9 (0 active connections)
Feb  4 02:30:27 hostname piler-smtp[1144]: connected from
192.168.165.154:41139 on fd=6 (active connections: 1)
Feb  4 02:32:18 hostname piler-smtp[1144]: client 192.168.165.154 
timeout

Feb  4 02:32:18 hostname piler-smtp[1144]: disconnected from
192.168.165.154 on fd=6 (0 active connections)
Feb  4 02:32:18 hostname piler-smtp[1144]: Removing 6KHX6ZMY7OD1R0KG
Feb  4 02:32:18 hostname piler-smtp[1144]: received: #013}^?,
from=, size=4940, client=192.168.165.154, fd=6
Feb  4 02:32:18 hostname piler-smtp[1144]: ERROR: couldn't rename
#013}^? to 2/ #013}^? (reason: no such file or
directory)
Feb  4 02:32:18 hostname piler-smtp[1144]: got signal: 11, Segmentation 
fault
Feb  4 02:32:18 hostname piler-smtp[1144]: piler-smtp has been 
terminated


Feb  5 02:32:15 hostname piler-smtp[28329]: connected from
192.168.165.252:57818 on fd=6 (active connections: 1)
Feb  5 02:32:15 hostname piler-smtp[28329]: connected from
192.168.165.252:57820 on fd=9 (active connections: 2)
Feb  5 02:32:15 hostname piler-smtp[28329]: disconnected from
192.168.165.252 on fd=6 (1 active connections)
Feb  5 02:32:15 hostname piler-smtp[28329]: disconnected from
192.168.165.252 on fd=9 (0 active connections)
Feb  5 02:33:15 hostname piler-smtp[28329]: connected from
192.168.165.252:60458 on fd=6 (active connections: 1)
Feb  5 02:33:15 hostname piler-smtp[28329]: connected from
192.168.165.252:60460 on fd=9 (active connections: 2)
Feb  5 02:33:15 hostname piler-smtp[28329]: disconnected from
192.168.165.252 on fd=6 (1 active connections)
Feb  5 02:33:15 hostname piler-smtp[28329]: disconnected from
192.168.165.252 on fd=9 (0 active connections)
Feb  5 02:34:15 hostname piler-smtp[28329]: connected from
192.168.165.252:34840 on fd=6 (active connections: 1)
Feb  5 02:34:15 hostname piler-smtp[28329]: connected from
192.168.165.252:34842 on fd=9 (active connections: 2)
Feb  5 02:34:15 hostname piler-smtp[28329]: disconnected from
192.168.165.252 on fd=6 (1 active connections)
Feb  5 02:34:15 hostname 

Re: Piler 1.3.10 issues with imap importing (Lotus Domino Server) from GUI

[sj]

Hello Hugo,

thank you for the pacthes. Have you tried to use imapfetch.py with 
python3?
Also can you track what happens or at what point pilerimport stucks? 
Anything

relevant in the mail logs?

I've just re-run the import test, and pilerimport has properly updated 
the

import table.

Janos

On 2021-01-25 13:27, Hugo Alvarez wrote:


We have installed Piler 1.3.10 and we cannot get imap importing (Lotus
Domino server) from GUI. Here is what we have been able to trackdown
so far:

1 - In read_folder_list we had to decode to utf-8  the case for the
tupple type. A type missmatch error would occur otherwise:

elif isinstance(folder, type(())):
folder = re.sub(r'\{\d+\}$', '',
folder[0].decode('utf-8')) + folder[1].decode('utf-8')

2 - Domino's Imap server returns a backslash the field separator:

[b'(\\Noinferiors \\HasNoChildren) "" Inbox', b'(\\Noinferiors
\\HasNoChildren) "" Trash', b'(\\HasNoChildren) "" Drafts',
b'(\\HasNoChildren) "" Sent']

Which does not match against the exiting regexp in read_folder_list in
imapfetch.py. This is fixed adding the backslash to the regexp:  f =
re.split(r' \"[\/\.\\]+\" ', folder)

3 - We are using python 2.7.5 under Centos 7.9.2009 which seens to
have a bug in imaplib's _checkquote(), which prevents folder names
with special characters from being correctly escaped making
imapfetch.py to fail. We fixed this by using a  sane _checkquote()
version from a previous imaplib. Maybe there is a cleaner solution ...

result.append(_checkquote(conn, f[1]))

4 - In process_folder, empty emails will fail when trying to write to
disk. This is fixed by checking for a tupple:

for num in data[0].split():
rc, data = conn.fetch(num, '(RFC822)')
if opts['verbose']:
print(rc, num)
if isinstance(data[0], tuple):
opts['counter'] += 1
with open("{}.eml".format(opts['counter']), "wb") as
f:f.write(data[0][1])

We could make it work with these changes, but pilerimport -T does not
seem to mark the fields 'updated' or 'finished' in the import table,
so subsequent executions of import.sh starts the whole process again
(failing the import due to duplicates).

The future of centos

[sj]

Dear piler users,

I'm aware of that many of you use centos 6, 7 and 8. And some systemd 
un-fans
even centos 5. I assume all of you have heard the recent news that 
centos 8
deceases at the end of this year as we know it, and Redhat suggests to 
move
to Centos Stream: 
https://blog.centos.org/2020/12/future-is-centos-stream/


Jim Salter of ArsTechnica wrote an in-depth article about what happened:
https://arstechnica.com/gadgets/2020/12/centos-shifts-from-red-hat-unbranded-to-red-hat-beta/

He also suggested some possible solutions and alternatives:
https://arstechnica.com/gadgets/2020/12/centos-linux-is-gone-but-its-refugees-have-alternatives/

I think some users will stay on centos 7 as long as possible (EOL status 
at the end of 2024).
Some users might go with the flow and switch to centos stream. And some 
other

users may migrate to something else.

I wonder what your opinion on the matter is.

Janos

piler 1.3.10 has been released

[sj]

Dear piler users,

new year, new release. I've just released version 1.3.10. The release 
notes document has 3 items:

- switching from Blowfish to AES-256
- obsoleted the tcp_wrappers with a postscreen style smtp acl list, see 
my previous email and

  https://www.mailpiler.org/wiki/current:smtp-acl-list
- added the security header feature

I've already covered the first two items. The security header is an 
optional feature allowing

you to specify a security header in piler.conf, eg.

security_header=X-My-Security-Header: a

Then if the first email header in the given message doesn't contain the 
header configured above,
then the message is discarded by the piler daemon. Note that the 
pilerimport utility disables
this feature, so you can safely import legacy emails not having such 
extra header line.


By default this feature is disabled. It's primary purpose is to provide 
a security mechanism
against unwanted email in the archive if limiting smtp clients via an 
IP-address list is not feasible.


Download: 
https://bitbucket.org/jsuto/piler/downloads/piler-1.3.10.tar.gz

SHA256: 55a70f94d0673d428ce433a12bdc49ee973cbd66eb3ffccf826d027383dc0eda

I also created a deb package for Ubuntu 20.04:
https://bitbucket.org/jsuto/piler/downloads/piler_1.3.10-focal-7fc8f74_amd64.deb
SHA256: 820063e005c211ddbc8bdaebec78f9a8bf19c7171d6b64d1fa0be897b04b7e13

There's even more, a docker image: sutoj/piler:1.3.10

Also, I've started to write a(n archiving related) blog, see 
https://mailpiler.com/blog/
Let me know if there's a topic you are interested in. For starters let 
me suggest two articles.


How to get an automatically renewing A-grade Let's Encrypt certificate 
with the traefik edge router:

https://mailpiler.com/how-to-obtain-an-a-grade-lets-encrypt-certificate-with-traefik/

How to run piler in a container using docker-compose:
https://mailpiler.com/running-piler-enterprise-in-a-containerized-environment-with-docker-compose/

The latter article focuses on the commercial edition, however with a 
little tweak you can use it

to run the docker image built with the open source edition.

I wish all of you a happy new year, stay safe!

Janos

A new feature: smtp acl rules

[sj]

Dear piler-users,

I've been using tcp_wrappers for limiting network access to several 
daemons,

and piler-smtp also supports it.

Unfortunately the redhat/centos distros have dropped tcp_wrappers 
support
starting with version 8, and in some cases there might be a better way 
to

do the job.

If you used postfix, then you might be familiar with postscreen, a nice 
tool

to block spammers on the smtp level. It also supports a cidr block based
access list that has inspired me to implement it for piler.

So piler starting with version 1.3.10 supports a similar concept, and 
has

dropped support for tcp_wrappers.

Note that 1.3.10 hasn't been released yet, give me a few days for that,
however the master branch is pretty close, and it already has this 
feature.


I've created a wiki page describing the feature in more details with 
examples:

https://www.mailpiler.org/wiki/current:smtp-acl-list

Note that it's an optional feature, it's your call whether you want or 
need it,
there might be other ways to restrict smtp access to the archive, eg. 
host based

iptables rules, a network based firewalls.

As always I appreciate your feedback. Also since it's almost the end of 
the year

I'd like to gather your ideas how to improve piler.

I wish all of you a happy new year!

Janos

Re: 1.3.9+ and CJK support in sphinx.conf

[sj]

Hello,

On 2020-12-27 08:31, d tbsky wrote:

   I want to upgrade our piler from "1.3.4 + sphinx 2.2" to "1.3.9/10
+ sphinx 3.3.1".
  I notice there is a new config entry at sphinx.conf:
"SPHINX_CHARSET_TABLE".I am still confused after reading the document.
   if I already setup "ngram_len" and "ngram_chars"  as FAQ said, do I
need to change the setting of "SPHINX_CHARSET_TABLE" of piler default
to something else?
   thanks a lot for your help!!


the SPHINX_CHARSET_TABLE settings cover the characters that most Western
countries use. If you want support for CJK languages, then you need to
uncomment ngram_len and ngram_chars in the line of NGRAM_CONFIG.

However, frankly I don't have much experience with CJK stuff, so I count
on your and other's experience and feedback whether the mentioned setup
works or if it needs some improvement.

Janos

Re: sphinx reindex question

[sj]

Hello,

On 2020-12-28 08:56, d tbsky wrote:


I was testing master commit fb9150f and upgrade from sphinx 2.2 to
3.3.1. according to piler manual, I need to reindex everything.
   so I install a new machine.  copy /var/piler and /var/lib/mysql  to
new machine and do procedure below:

  rm -f /var/piler/sphinx/*
  runuser -u piler -- indexer --all --config /etc/piler/sphinx.conf
  cd /var/piler/tmp
  runuser -u piler -- reindex -a  (why this command didn't need to
specify /etc/piler/sphinx.conf?)


Because it has nothing to with sphinx. The reindex tool retrieves the
emails, parses them, and puts their contents to the sph_index table.
Then the sphinx indexer reads and process this table.


 I found I can not understand what happened.
 the size at old system:
 /var/piler/sphinx => 300MB
/var/lib/mysql/piler/sph_index.ibd => 22M

the size at new system after reindex:
/var/piler/sphinx => 89M, which didn't change at all at the reindex 
process


sphinx 3.3.1 has some improvements over the 2.2 series. If you can find
all your email, then don't worry.


/var/lib/mysql/piler/sph_index.ibd => become 1.9GB


This is the before mentioned table. Since you put a huge volume of data
during the reindex process this table has grown, and now it just doesn't
want to shrink. Try running "optimize table sph_index;" at the mysql
console to shrink it when the reindex has completed.



/var/piler/tmp => 17G, the same size as /var/piler/store

so the whole process seems extract mails from /var/piler/store to
/var/piler/tmp, and write to mysql.

I don't know is this behavior correct? why reindex touch nothing at
/var/piler/sphinx and why mysql db become so huge.

I try to open piler web UI and do some mail search, it seems working
fine, but I am still confused.


That's the ultimate test that you have done it well or not :-)

Janos

Re: Piler is not saving the incoming emails

[sj]

On 2020-12-14 15:37, jb wrote:


I only send you the lines from /var/log/mail.log. Do you wanted other 
ones to?


No. However, when the system boots and starts piler, then you usually 
have such

lines, eg.

Dec 13 16:59:17 cust1 piler-smtp[203]: reloaded config: 
/etc/piler/piler.conf

Dec 13 16:59:17 cust1 piler-smtp[203]: running as user 'piler'
Dec 13 16:59:17 cust1 piler-smtp[203]: created directory: 
*/var/piler/store/00*
Dec 13 16:59:17 cust1 piler-smtp[203]: created directory: 
*/var/piler/tmp/0*
Dec 13 16:59:17 cust1 piler-smtp[203]: created directory: 
*/var/piler/tmp/1*

Dec 13 16:59:17 cust1 piler-smtp[203]: piler-smtp 1.5.0-47e2e50 starting
Dec 13 16:59:17 cust1 piler[205]: reloaded config: /etc/piler/piler.conf
Dec 13 16:59:17 cust1 piler[205]: piler 1.5.0-47e2e50 starting

Anyway, if the systemd fix has solved it, it's ok for me.

Janos

Re: Piler is not saving the incoming emails

[sj]

On 2020-12-14 15:13, jb wrote:



piler.service:
[Unit]
Description=Piler daemon
After=network.target mariadb.service


...

The systemd files look fine.



Dec 14 15:09:01 piler piler-smtp[392]: connected from 1.2.3.4:38359
on fd=6 (active connections: 1)
Dec 14 15:09:01 piler piler-smtp[392]: got on fd=6: *EHLO
www.example.org#015#012* [1]
Dec 14 15:09:01 piler piler-smtp[392]: sent on fd=6:


250-piler.example.org#015#012250-PIPELINING#015#012250-STARTTLS#015#012250-SIZE#015#012250

8BITMIME#015
Dec 14 15:09:01 piler piler-smtp[392]: got on fd=6:
*STARTTLS#015#012*
Dec 14 15:09:01 piler piler-smtp[392]: starttls request from client
Dec 14 15:09:01 piler piler-smtp[392]: sent on fd=6: 220 Ready to
start TLS#015
Dec 14 15:09:01 piler piler-smtp[392]: waiting for ssl handshake
Dec 14 15:09:01 piler piler-smtp[392]: SSL_accept() result, rc=-1,
errorcode: 2, error text: error::lib(0):func(0):reason(0)
Dec 14 15:09:01 piler piler-smtp[392]: got on fd=6: *EHLO
www.example.org#015#012* [1]
Dec 14 15:09:01 piler piler-smtp[392]: sent on fd=6:


250-piler.example.org#015#012250-PIPELINING#015#012250-SIZE#015#012250

8BITMIME#015
Dec 14 15:09:01 piler piler-smtp[392]: got on fd=6: *MAIL
FROM: SIZE=246722#015#012*
Dec 14 15:09:01 piler piler-smtp[392]: sent on fd=6: 250 Ok#015


this is only a redacted stuff for piler-smtp, and not even complete for 
piler-smtp.
Anyway, the above entries look fine, you have received several emails 
via smtp.


I assume that later log entries show that piler daemon child processes 
have

processed these emails.



I think I found a fix when I change this line After=network.target
auditd.service piler.service in piler-smtp service it works. Can this
be true?


I don't think so. The piler daemon processes and piler-smtp run 
independently
from each other. It seems that piler-smtp runs fine, but piler has some 
issues.


That's why I asked all piler related mail logs (the first 2-300 lines at 
max)

since the last reboot. You clearly didn't give me that.

Anyway, if your systemd modifications prove to work, then I'm happy with 
that.

Keep it.


Janos

Re: Piler is not saving the incoming emails

[sj]

Hello Jonathan,

On 2020-12-14 13:48, jb wrote:


I run here the latest piler version, on debian 10 and I have the
problem, that after a restart piler is not saving anymore the emails.
I had this now multiple times, and I think it have to do with the
piler.pid file.

I saw now many time in systemctl status piler:


...

Dez 14 12:37:47 piler piler[530]: reloaded config:
/usr/local/etc/piler/piler.conf
Dez 14 12:37:47 piler piler[530]: piler 1.3.9, build 998 starting
Dez 14 12:37:47 piler systemd[1]: piler.service: Can't open PID file
/run/piler/piler.pid (yet?) after start: No such file or directory
Dez 14 12:37:47 piler systemd[1]: Started Piler daemon.


I try to change the path from pid file in piler.conf and in the
service file, and when I restart the service it can happen, that this
error disappear. But after next reboot it comes again.

On default path to /var/run/piler/piler.pid, the file would also
exists, so why the service is not finding the pid file, I don't know.

When emails are coming in, piler-smtp will notice it.

Have you any suggestions?


Yes. Show me your piler related systemd files. Then stop piler and 
piler-smtp,
and verify the pid files are removed, and using ps output that no piler 
daemon
process is running. Then set verbosity=5 in piler.conf, and reboot your 
host.
Finally, send a test email to the archive, and show me the mail log 
entries

since the reboot.

Also double check that you want to start piler only once at boot time 
(ie.

not with systemd and with rc.local or similar).

Janos

do you use the folder feature?

[sj]

Dear piler-users,

piler features the folder feature, and I got a request a few years ago
to revise and fix it (check out this old branch: 
https://bitbucket.org/jsuto/piler/branch/per_user_folder2),

and a new request recently to adjust it.

Before deciding whether to revise and fix the folder feature (and 
possible break it
for those who already use it) or to develop a new feature, perhaps 
called as buckets,

I'd like to know if you actually use the feature.

If so, please drop me a line. Or to the list.

Janos

updates on encryption

[sj]

Dear piler-users,

I'm writing to you about the encryption that piler uses. tl;dr: piler is 
switching from
Blowfish 128 bit to AES 256 bit encryption. It's a bit long and 
technical to some degree,
but please read it through to get the whole picture, and understand 
there's no need to be

afraid of.

Piler uses Blowfish encryption CBC mode. The piler.conf file has a 
parameter for the initalization
vector (iv) required for CBC. Later I've fixed the code that iv is empty 
by default.


No matter if iv is set to a fix value or it's empty (as it is nowadays 
by default) it allows some room for
an attacker to draw some conclusions and try to guess the contents of 
the encrypted files. Fortunately it's

not that bad as it may sound.

Firstly, the attacker needs to have physical access to the stored files 
in /var/piler/store directories.
Normally nobody has such access other than root and the piler user. Then 
piler compresses the data before
encrypting it which offers some redundancy to mitigate such 
(theoretical) attacks.


Frank Schmirler who reported the issue also provided a patch to improve 
the encryption. The idea is to prepend
a random block of data read from /dev/urandom to beginning of the real 
data to render such attempts ineffective.
Then when the file is retrieved from the archive piler checks if such a 
junk data block exists and if so it
discards it. It's completely transparent and backward compatible. The 
patch is already on the master branch.


However, there's more. The Blowfish encryption algorithm was developed 
in ~1993, so you might say it's a bit
out of date. However when the piler project started I've picked a time 
tested solution which I believe that
Blowfish was and still is. The algorithm offers the encryption key up to 
448 bits (=56 bytes), which is plenty

for a symmetric encryption algorithm.

The below article explains that even a 256 bit key is more than enough 
to render any brute force attack ineffective.
A quote from Bruce Schneier's Applied Cryptography book: 
https://security.stackexchange.com/questions/25375/why-not-use-larger-cipher-keys/25392#25392


So the piler.key is 448-bit long, and even a 256 bit key would be 
plenty. But what if you had only a 128-bit
encryption key? Would it be still enough? The trouble is that the 
openssl implementation of Blowfish is limited
to 128 bit by default, see 
https://security.stackexchange.com/questions/25393/openssl-blowfish-key-limited-to-256-bits/25405#25405

for more.

I was unaware of this behaviour of openssl until a few days ago when 
Frank educated me on the matter. Before panicking

please consider that a commenter at the above link says that

"Above 128 bits in unnecessary. With 128 bits there are 2^128 possible 
keys, divided by 100 billion tests per second
(which would require a formidable GPU farm) and it would take someone 
7.8*10^9 times the age of the universe to crack

it (about 10^20 years)."

So even if piler is limited to a 128-bit encryption key instead of the 
possible 448-bit, I personally still believe that
there's no reason to panic and drop your archive. Frank thinks, and I 
agree with him that Blowfish will not become insecure
in the near future. However, an email archive should last for a long 
time, and over the decades I cannot make any promises

or guarantee that 128 bit will be enough till the end of the world.

So I've switched from Blowfish 128-bit to AES-256 bit. This time openssl 
truly delivers a 256 bit encryption. The patch
I made uses Blowfish encryption to decrypt legacy emails in the archive. 
However, every new email is encrypted with AES-256.
The piler id has also changed. It starts with 4. for Blowfish 
encrypted emails, however it starts as 5000
to indicate that it's an AES-256 encrypted email, and piler decrypts it 
using the appropriate method figured from the
file name. Again, this patch is also on the master branch, and it's 
backward compatible. You may (and in fact must) keep piler.key

as it is.

Modern CPUs have hardware accelerators for AES-256, and it may run up to 
50% faster than Blowfish, so the switch is worth
even from a cpu performance perspective, though I believe that Blowfish 
is not a bottleneck at all during the processing

of emails.

Again, I thank Frank Schmirler for bringing this issue to my attention, 
and providing hints and even patches to alleviate the problem.


Let me know your opinions on the matter. I also invited Frank to comment 
and share his thought, too on the mailing list.



Janos

healthcheck tool

[sj]

Dear piler users,

while I was checking the product mailstore (an email archiving 
competitor)

what features they offer, I found telemetry. The mailstore telemetry is
an opt-in feature sends some key metrics allowing the mailstore company
to better understand the usage of their product. The feature is alleged
to be completely transparent, and the data to be sent is human readable.

I'm not sure if you opted in for such feature for piler, however a
similar tool providing insights about the internal state of the archive
host would definitely be useful. So I've just committed a healthcheck
utility to the master branch, feel free to use it. It collects some
basic metrics, eg. cpus, memory, disk utilization, piler processes,
sphinx and mysql db sizes, piler's crontab entries.

Note that it doesn't send anything to anywhere, it prints its output
to stdout only. In case you have an issue it might be handy to have
the most important data on your screen giving some clues where to
start the troubleshoot.

You may get the tool at 
https://bitbucket.org/jsuto/piler/src/master/util/healthcheck.sh

and you can just download and use it for virtually any version of piler
(no need to upgrade just for the tool).

Janos

Re: Piler 1.3.5 on Ubuntu 18.04 WebUI doesn't render timestamped Mails

[sj]

Hello Patrick,

it seems that a few things have changed, so I've fixed the issue
to make the feature work again.

Please apply the following commits:

https://bitbucket.org/jsuto/piler/commits/4a131f3058766ecdeb3021f5478746f58938c2c8
https://bitbucket.org/jsuto/piler/commits/427114b89202e3658664540e2eb5d30feeba659b

OR

simply overwrite 
https://bitbucket.org/jsuto/piler/src/master/util/sign.php
and 
https://bitbucket.org/jsuto/piler/src/master/webui/system/helper/TrustedTimestamps.php

then add the following to /var/piler/www/config.php:
$config['TSA_VERIFY_CERTIFICATE'] = true;


Janos

PS: Thank you Frank for the freetsa.org hint. It works properly.


On 2020-10-17 12:45, Wagner, Patrick wrote:

Hello Janos,

I've changed the DECRYPT_ATTACHMENT_BINARY to pileraget and I've
re-read the two links you sent, but are not sure I could've missed
anything?

For the packaged 1.3.5 release, is there more to do than to add the
four TSA_XXX config lines and to add a regular cronjob to execute
sign.php as piler user?

"Something" is definitely happening, as I can read mails via web UI
just fine, until 500 non-stamped mails have been reached, where
suddenly a batch of 500 e-mails is no longer readable.

For what it's worth, here's my entry taken from the piler user crontab:

0 * * * *   /usr/bin/php /usr/libexec/piler/sign.php --webui
/var/piler/www --mode unit

And here's what happens if I run it manually:

piler@mypiler01:~$ /usr/bin/php /usr/libexec/piler/sign.php --webui
/var/piler/www --mode unit
PHP Warning:  Use of undefined constant COUNT - assumed 'COUNT' (this
will throw an Error in a future version of PHP) in
/usr/libexec/piler/sign.php on line 3
PHP Warning:  Use of undefined constant HASH_VALUE - assumed
'HASH_VALUE' (this will throw an Error in a future version of PHP) in
/usr/libexec/piler/sign.php on line 4
PHP Warning:  Use of undefined constant RESPONSE_STRING - assumed
'RESPONSE_STRING' (this will throw an Error in a future version of
PHP) in /usr/libexec/piler/sign.php on line 5
PHP Warning:  Use of undefined constant RESPONSE_TIME - assumed
'RESPONSE_TIME' (this will throw an Error in a future version of PHP)
in /usr/libexec/piler/sign.php on line 6
PHP Warning:  Use of undefined constant START_ID - assumed 'START_ID'
(this will throw an Error in a future version of PHP) in
/usr/libexec/piler/sign.php on line 7
PHP Warning:  Use of undefined constant STOP_ID - assumed 'STOP_ID'
(this will throw an Error in a future version of PHP) in
/usr/libexec/piler/sign.php on line 8

PHP Warning:  ini_set(): Headers already sent. You cannot change the
session module's ini settings at this time in
/var/piler/www/config.php on line 333
PHP Warning:  session_start(): Cannot start session when headers
already sent in /var/piler/www/system/request.php on line 30
piler@mypiler01:~$ echo $?
0

Should I be using a different PHP version or are those warnings to be 
expected?


FreeTSA should be available for your testing purposes.


Thanks,

Patrick

On 16.10.2020 21:19, s...@acts.hu wrote:



Hello Patrick,

for starters the DECRYPT_ATTACHMENT_BINARY should be pileraget (not 
pilerget).


Also it might be worth to go over the steps and check if everything is 
setup properly:


https://bitbucket.org/jsuto/piler/issues/480/support-rfc3161-trusted-timestamps 
https://www.mail-archive.com/piler-user@list.acts.hu/msg00785.html


In the meantime I try to find a TSA provider suitable for testing 
purposes.
If anyone on the list was familiar with any, or better could provide a 
test

account, it would be great.

Janos



On 2020-10-16 20:02, Patrick Wagner wrote:

Hello everyone,

we're testing the TSA signing feature of Piler. When I login as
auditor all mails are listed correctly in the Web GUI upper pane /
list. When I click on a mail the header and content are displayed
below if that mail was not signed (yet). Clicking on a signed mail
however does not refresh the lower pane and continues to display
either the last non-stamped mail or remains blank (if no mail had 
been

selected before). With pilerget on the CLI the mails are displayed
correctly.

What's wrong? Do I have to change any configuration?

I added these lines in config-site.php:
$config['TSA_PUBLIC_KEY_FILE'] = '/etc/piler/tsa.publickey.pem';
$config['TSA_START_ID'] = 1;
$config['TSA_STAMP_REQUEST_UNIT_SIZE'] = 500;
$config['TSA_URL'] = 'http://zeitstempel.dfn.de';

$config['DECRYPT_BINARY'] = '/usr/bin/pilerget';
$config['DECRYPT_ATTACHMENT_BINARY'] = '/usr/bin/pilerget';


Thanks,
Patrick

# piler -V
piler 1.3.5, build 997, Janos SUTO 

Build Date: Sun Apr 21 16:50:30 UTC 2019
ldd version: ldd (Ubuntu GLIBC 2.27-3ubuntu1) 2.27
gcc version: gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)
OS: Linux 188ae4f9894f 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13
10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Configure command: ./configure --prefix=/usr --sysconfdir=/etc
--localstatedir=/var --with-database=mariadb
MySQL client library version: 10.2.9

Re: Piler 1.3.5 on Ubuntu 18.04 WebUI doesn't render timestamped Mails

[sj]

Hello Patrick,

for starters the DECRYPT_ATTACHMENT_BINARY should be pileraget (not 
pilerget).


Also it might be worth to go over the steps and check if everything is 
setup properly:


https://bitbucket.org/jsuto/piler/issues/480/support-rfc3161-trusted-timestamps
https://www.mail-archive.com/piler-user@list.acts.hu/msg00785.html

In the meantime I try to find a TSA provider suitable for testing 
purposes.
If anyone on the list was familiar with any, or better could provide a 
test

account, it would be great.

Janos



On 2020-10-16 20:02, Patrick Wagner wrote:

Hello everyone,

we're testing the TSA signing feature of Piler. When I login as
auditor all mails are listed correctly in the Web GUI upper pane /
list. When I click on a mail the header and content are displayed
below if that mail was not signed (yet). Clicking on a signed mail
however does not refresh the lower pane and continues to display
either the last non-stamped mail or remains blank (if no mail had been
selected before). With pilerget on the CLI the mails are displayed
correctly.

What's wrong? Do I have to change any configuration?

I added these lines in config-site.php:
$config['TSA_PUBLIC_KEY_FILE'] = '/etc/piler/tsa.publickey.pem';
$config['TSA_START_ID'] = 1;
$config['TSA_STAMP_REQUEST_UNIT_SIZE'] = 500;
$config['TSA_URL'] = 'http://zeitstempel.dfn.de';

$config['DECRYPT_BINARY'] = '/usr/bin/pilerget';
$config['DECRYPT_ATTACHMENT_BINARY'] = '/usr/bin/pilerget';


Thanks,
Patrick

# piler -V
piler 1.3.5, build 997, Janos SUTO 

Build Date: Sun Apr 21 16:50:30 UTC 2019
ldd version: ldd (Ubuntu GLIBC 2.27-3ubuntu1) 2.27
gcc version: gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)
OS: Linux 188ae4f9894f 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13
10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Configure command: ./configure --prefix=/usr --sysconfdir=/etc
--localstatedir=/var --with-database=mariadb
MySQL client library version: 10.2.9
Extractors: /usr/bin/pdftotext /usr/bin/catdoc /usr/bin/catppt
/usr/bin/xls2csv /usr/bin/unrtf /usr/bin/tnef

Ubuntu 18.04.5 LTS

PHP 7.2.24-0ubuntu0.18.04.6 (as supplied with the distro)

MariaDB 10.1.44 (as supplied with the distro)

Re: Using AD Accounts

[sj]

Hello Stefan,

On 2020-09-17 10:52, Stefan Günther wrote:


I have a general question abou using AD accounts in mailpiler.

After configuring the AD authentication, I can successfully log into
the web interface with one the AD users.

My expectation was that this user should automatically be able to
search for emails with his address as recipient, but the search does
not return the two messages I have send about a day ago.

Do I have to add every account that should be able to search for
personal emails to the auditor group?


surely no. When you log in the gui, check your email addresses in the
upper right corner settings menu. If your emails are not there you
need to fix the email attribute the gui uses to query your email 
addresses.


You may also check the sphinx query in the mail log to see what email
addresses the gui assigned to you.

Janos

Re: exact matching for archiving rules possible?

[sj]

On 2020-09-10 11:16, Martin Nadvornik wrote:


thank you as always for your quick help. You were absoluteley right,
using the two rules works perfectly.
In case someone stumbles upon this in future and thinks two rules are
too unwieldy, the regex "(^someth...@example.com)|(
someth...@example.com)" also works fine and keeps it in one rule.


that's even better :-) I'll add it to the docs.

Janos

Re: exact matching for archiving rules possible?

[sj]

Hello Martin,

On 2020-09-09 14:33, Martin Nadvornik wrote:


I recently discovered that archiving rules like "From:
someth...@example.com" or "To: someth...@example.com" will also trigger
for e-mail addresses like "justsometh...@example.com". Is there any way
to perform an exaxct matching?
I already tried entering the following regexes when creating the rules
via the web gui:

"someth...@example.com"
^ someth...@example.com
^someth...@example.com

Any help is appreciated :)


Take a message as an eml file, and run pilertest against it to see how
piler sees the message. Only showing the To: results between asterisks 
(*).


Let's take the following message (#1) (only the To: header):

To: user.n...@example.com

after parsing:

to: *user.n...@example.com user name example com  (example.com )*


Then another message (#2):

To: "User, Name" user.n...@example.com

after parsing:

to: *user name user.n...@example.com user name example com  (example.com 
)*



If you specified "^user.n...@example.com" (without quotes), then you 
could match

message #1, but miss #2.

If you specified "user.n...@example.com" (without quotes), then you 
could match
both message #1 and #2, however also match otheruser.n...@example.com, 
which you

don't want.

So the solution takes 2 patterns:

"^user.n...@example.com" to catch #1

and

" user.n...@example.com" to catch #2 (notice the prepended space!)

Again, you shouldn't use the quotes when you type these patterns, the 
quotes

here only serve as a clear distinction.

Janos

Re: Piler 1.3.9 / Sphinx 3.3.1

[sj]

Hello,

get the latest master, it fixes the issue.

Janos


On 2020-08-29 21:49, BKH-Netz.de - Postfach wrote:

Hello Janos,

on a completely new installation with Piler 1.3.9 / Sphinx 3.3.1 and
with no imported mails i alway get this error:

ERROR: index 'delta1': reserved field name 'from'.
FATAL: failed to merge index 'delta1' into index 'dailydelta1': failed
to open /var/piler/sphinx/dailydelta1.sph: No such file or directory

Is this „normal“ because there are no mails in sphinx?

On Sphinx 3.2.1 and lower this error never occured on a completely new
installation.

Regards

Re: pilerimport from IMAP-server with STARTLS

[sj]

I suggest to fix imapfetch.py.

Locate this part in the file, and fix the port number:

if args.port == 993:
conn = imaplib.IMAP4_SSL(args.server)
else:
conn = imaplib.IMAP4(args.server)

Janos


On 2020-08-27 10:36, Yann Lehmann wrote:

Am 27.08.2020 um 10:09 schrieb s...@acts.hu:



yes, use the -p 993, and pilerimport will automatically try using tls.

Or you may even try the util/imapfetch.py tool to do the same. Note 
that
imapfetch.py only downloads emails. Then you need to process the 
downloaded

emails with pilerimport.


Janos



Thank you Janos for the very quick response. Our server uses STARTTLS
on port 143, so I tried with -P 143 but with the same result as
without specifying a port.

Obviously, specifying -P 993 doesn't work either.

Re: pilerimport from IMAP-server with STARTLS

[sj]

Hello,

On 2020-08-27 09:57, Yann Lehmann wrote:


Due to a server dysfunction of our mail-server, our piler server could
not archive mails for several weeks.

As most of our users do not delete mails from their inbox very often,
there is a good chance that most of them could be imported.

However, when I try to import from our IMAP-server (pilerimport -i
server -u username -p password), I get a response, that plaintext
without SSL/TLS is not allowed.

Is there a way to connect to the IMAP-server with SSL/TLS enabled ?



yes, use the -p 993, and pilerimport will automatically try using tls.

Or you may even try the util/imapfetch.py tool to do the same. Note that
imapfetch.py only downloads emails. Then you need to process the 
downloaded

emails with pilerimport.


Janos

Release of 1.3.9, and support for sphinx-3.3.1

[sj]

Dear piler users,

I've just released version 1.3.9 of piler. You may find below the 
changes at the
end of this email. I'd like to describe the sphinx related changes in a 
bit more

detail.

Sphinx 3.3.1 introduced a strict check on the sql query for the indexer. 
Instead
of relying on the underlying sql engine (mysql in our case) to determine 
if the
sql query is syntactically correct, Sphinx took the matter to its own 
hands, and

rejecting the default piler query, even though it's correct for mysql.

Since I haven't got any meaningful response about the issue on the 
sphinx forum

in a month, we have two choices:

a) don't upgrade above sphinx-3.2.1. Sphinx 3.2.1 seems to be the last 
version

supporting the current sql query piler uses.

b) eat the bullet (switch to strict schema, and reindex everything).

The idea of sticking with a certain version of a product, and never 
upgrade cringes

me, so I think sooner or later I need to take option b).

To do that you need to do the following steps:
- upgrade to piler 1.3.9
- set $config['SPHINX_STRICT_SCHEMA'] = 1; in config-site.php
- set define('SPHINX_STRICT_SCHEMA', 1); in sphinx.conf
- reindex everything

I'll add a separate section for the sphinx 3.3.1 related stuff to the 
online documentation

with the above info.

The download link: 
https://bitbucket.org/jsuto/piler/downloads/piler-1.3.9.tar.gz
(SHA256: 
eed041507fd9b16f59002570fde34b004980be70362efdec6f90b5f05face867)




Also I've neglected documenting the release notes since 1.3.0, so it's 
high time to

continue tracking what's changed. These are the 1.3.9 changes:


- Added a separator to searching for attachment names
- [BUGFIX] Render multiple mail parts in mail view instead of only the 
last part
- Use TLS v1.2 with openssl 1.0.x for connecting remote pop3/imap 
servers

- Instant search results to the gui when the search page loads
- Support sphinx-3.3.1, introduced sphinx strict mode variable
- GUI domain fixes
- gcc 9 fixes
- Fix permission on sphinx data dir to 700
- pilerpurge.py should honor the mysqlhost value
- Password change enabled by default
- Health page fixes
- GUI mime parser fixes
- Start/stop script fix
- Optimized search page for mobile devices, set 
$config['ENABLE_MOBILE_PREVIEW'] = 1; in config-site.php to enable it



Janos

Re: Piler 1.35 on Ubuntu 18.04 - Web UI doesn't show mail headers / contents

[sj]

On 2020-07-31 17:16, Wagner, Patrick wrote:


So that issue is fixed now, but has unearthed another - I can find and
open all mails just fine as auditor@local, and logging in with my
personal account works as well (I'm shown only my emails, as it should
be), but another email address / account I've tested doesn't show any
emails even though the usage report and my login as auditor definitely
contains emails for this email address? I've even used copy and paste
out of the usage report UI to change email address and username for
the piler account in question (basically a NOOP, as there was no typo
before either)


check the sphinx query logged when this other user hits the search 
button.
Verify that his email addresses are present in the query. If that's 
good,

then download or export a single email he should see, and run pilertest
on the email to see what piler thinks about it. Check that the 
From/To/Cc

fields have his email address.

Janos

Re: Deleting emails

[sj]

Hello Marcelo,

connect to the piler mysql database, and execute the following query:

update metadata set retained=10 where id > x;

Then let pilerpurge.py take care of the rest.

x: is the numeric id of the last email you want to keep.
Make sure to double check, because after purging the emails
they are gone for good.

Janos

On 2020-07-22 00:49, Marcelo Machado wrote:

Hello everyone.

How to delete emails from a specific date until the end of all stored
emails?

Marcelo Gomes

Re: Enterprise edition

[sj]

It's ok. Btw. thanks Martin reporting the website issue. The site's 
cache

was corrupted, now it's back online.

Janos

On 2020-07-22 21:17, Marcelo Machado wrote:

Janos, I'm sorry if I was misunderstood.
I didn't know the piler.com [1] website and I accidentally entered it
thinking I was accessing the piler.org [2] website.
It was a mess on my part so I asked the question.
Sorry if I caused a problem.

Marcelo

Em qua., 22 de jul. de 2020 às 16:08,  escreveu:


I'd like to clarify a few things. I'll use the following conventions
in this email to distinguish between the open source and commercial
editions of piler:

- 'piler': the open source email archiving project
- 'piler-ee': the commercial, enterprise version of piler

I'm a huge fan of open source projects, and even though I'm working
on a
paid, commercial product of piler-ee, I'd like to emphasize that

- I intend to keep piler open source, you have access to the source
code
- I won't remove any feature from piler to make piler-ee more
attractive
- I won't force you to switch to piler-ee*
- You can use piler for whatever purposes absolutely free of charge

Note that despite piler doesn't have the word 'enterprise' among its

adjectives,
it doesn't mean it's a crappy product you shouldn't use it for
company
purposes. It's quite the opposite. I believe that piler is a decent
product
for a secure email archive (especially if you regularly backup its
key
components,
but that's a story for another day).

However, I've been looking for ways to monetize the popularity of
piler,
so I've decided to fork it, and been working on piler-ee, including
commercial
services as well even for piler.

*: check out https://mailpiler.com/ if you are curious what piler-ee
offers for you.

Janos

On 2020-07-22 18:07, Martin Nadvornik wrote:

Hi Marcelo,

can't speek for Janos, the developer of Piler, but I am not aware

of

any changes. What did you read that leads you into thinking piler

is

not open source anymore?
The source code is available like always at
https://bitbucket.org/jsuto/piler/src/master/ and the master

branch

had the last commit 4 days ago...

There has always been an enterprise edition (at least as long as I
know the project) but you are not required to pay anything to use
piler when using the open source version.

Or maybe you got confused because the site currently seems a bit
broken and clicking on "GoBD notes" leads to the enterprise
documentation. The normal documentation ist still reachable under
http://www.mailpiler.org/wiki/current:index however.

Janos: FYI the website seems broken. I guess "Please fill or

disable

this placeholder" is not something that is supposed to be there :)

Regards,
Martin

Am 2020-07-22 17:51, schrieb Marcelo Machado:

Hello everyone.

Is Mail Piler only commercial now?

Does the Open Source Edition still exist?

Marcelo Gomes



Links:
--
[1] http://piler.com
[2] http://piler.org

Re: Enterprise edition

[sj]

I'd like to clarify a few things. I'll use the following conventions
in this email to distinguish between the open source and commercial
editions of piler:

- 'piler': the open source email archiving project
- 'piler-ee': the commercial, enterprise version of piler


I'm a huge fan of open source projects, and even though I'm working on a
paid, commercial product of piler-ee, I'd like to emphasize that

- I intend to keep piler open source, you have access to the source code
- I won't remove any feature from piler to make piler-ee more attractive
- I won't force you to switch to piler-ee*
- You can use piler for whatever purposes absolutely free of charge


Note that despite piler doesn't have the word 'enterprise' among its 
adjectives,

it doesn't mean it's a crappy product you shouldn't use it for company
purposes. It's quite the opposite. I believe that piler is a decent 
product
for a secure email archive (especially if you regularly backup its key 
components,

but that's a story for another day).

However, I've been looking for ways to monetize the popularity of piler,
so I've decided to fork it, and been working on piler-ee, including 
commercial

services as well even for piler.

*: check out https://mailpiler.com/ if you are curious what piler-ee
offers for you.


Janos


On 2020-07-22 18:07, Martin Nadvornik wrote:

Hi Marcelo,

can't speek for Janos, the developer of Piler, but I am not aware of
any changes. What did you read that leads you into thinking piler is
not open source anymore?
The source code is available like always at
https://bitbucket.org/jsuto/piler/src/master/ and the master branch
had the last commit 4 days ago...

There has always been an enterprise edition (at least as long as I
know the project) but you are not required to pay anything to use
piler when using the open source version.

Or maybe you got confused because the site currently seems a bit
broken and clicking on "GoBD notes" leads to the enterprise
documentation. The normal documentation ist still reachable under
http://www.mailpiler.org/wiki/current:index however.

Janos: FYI the website seems broken. I guess "Please fill or disable
this placeholder" is not something that is supposed to be there :)

Regards,
Martin

Am 2020-07-22 17:51, schrieb Marcelo Machado:

Hello everyone.

Is Mail Piler only commercial now?

Does the Open Source Edition still exist?

Marcelo Gomes

Re: Empty message header and body

[sj]

On 2020-07-16 21:35, Marcelo Machado wrote:

Just an observation.
The migration documentation on the website does not mention anything
about the piler.key file.


"Make sure that piler.key on NEW is the same as on OLD, and if the 'iv'
parameter is set on OLD in piler.conf, it must have the same value on 
NEW."


See http://www.mailpiler.org/wiki/current:migration-to-new-host for 
more.


Janos

Re: Empty message header and body

[sj]

On 2020-07-16 20:31, Marcelo Machado wrote:

- is there an iv parameter in the old host's piler.conf file?
No.

- are the piler.key files on old and new host identical?

Not also.


That's the problem. Legacy emails can be decrypted with old piler.key 
only.

However, you started to encrypt new emails with a different key.

I suggest you to stop piler, then export all new emails, and start over
the migration process. Be sure to use piler.key from old host.

Verify that you can access legacy emails on the new host, and then
import the exported emails.

Janos

Re: Empty message header and body

[sj]

Usually I don't require any sensitive information to troubleshoot,
eg. your mysql password, or even the attached encryption key.

Anyway, I can see that no 'iv' parameter in piler.conf on the
new host. However,

- is there an iv parameter in the old host's piler.conf file?
- are the piler.key files on old and new host identical?

Janos

On 2020-07-16 00:42, Marcelo Machado wrote:

In attachment is my piler.key

Below is my piler.conf (I changed the domain):

archive_emails_not_having_message_id=0
archive_only_mydomains=0
backlog=20
check_for_client_timeout_interval=20
cipher_list=ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
clamd_socket=/tmp/clamd
default_retention_days=90
enable_chunking=0
enable_cjk=0
enable_folders=0
encrypt_messages=1
extract_attachments=1
extra_to_field=X-Envelope-To:
helper_timeout=20
hostid=arch...@piler.mydomain.com.br
listen_addr=0.0.0.0
listen_port=2525
max_connections=64
max_requests_per_child=1
memcached_servers=127.0.0.1
memcached_to_db_interval=900
memcached_ttl=86400
min_message_size=100
min_word_len=1
mmap_dedup_test=0
mysqlcharset=utf8mb4
mysql_connect_timeout=2
mysqldb=piler
mysqlpwd=n3ts0l
mysqlsocket=/var/run/mysqld/mysqld.sock
mysqluser=piler
number_of_worker_processes=2
pemfile=/usr/local/etc/piler/piler.pem
pidfile=/opt/run/piler/piler.pid
piler_header_field=X-piler-id:
process_rcpt_to_addresses=0
server_id=0
smtp_timeout=60
spam_header_line=X-Spam-Flag: YES
syslog_recipients=0
tls_enable=1
tweak_sent_time_offset=0
update_counters_to_memcached=0
username=piler
verbosity=1
workdir=/opt/piler/tmp
queuedir=/opt/piler/store

Em qua., 15 de jul. de 2020 às 08:33, Janos SUTO 
escreveu:


Sorry, piler.key is a file, not an entry in piler.conf.

Janos
On 15 Jul 2020, at 12:25, Marcelo Machado 
wrote:

Thanks for the answer.

The parameter iv is cipher_list.There is no line with piler.key.
I was unable to read the contents of piler.key.

Marcelo

Em qua., 15 de jul. de 2020 às 00:46, Janos SUTO < s...@acts.hu>
escreveu:

Hello Marcelo,

check if piler.key and the iv parameter in piler.conf (if there's
any) are the same.

Janos
On 14 Jul 2020, at 23:42, Marcelo Machado < mmgom...@gmail.com>
wrote:

Hello everyone.

After I migrated MailPiler to a new hardware, the header and body of
old messages no longer appear. The new ones are OK.

I followed the migration documentation on the website.

Marcelo Gomes


piler.key
Description: Binary data

Re: Rcpt Table Question

[sj]

Hello Ryan,

On 2020-05-03 00:15, Ryan Blenis wrote:


Thanks for the quick reply. Yes they're all different... but very
oddly formatted.

I can send you a copy of the output directly if you'd like, as I don't
want to expose a bunch of emails. But the gist is that many have
periods, dashes, or numbers preceding the actual email address, in
some cases as odd as "--@--.com", "@hotmail.com [2]",
"---@hotmail.com [2]", "-@..immature", "-@..khairy",
"-@d.i.s.c.o", "..@www.hip",
"..redactedhotmailaddr...@hotmail.com..me",
"._.pitch@n._", etc.


indeed, they are clearly bogus addresses.



I did a pilerget on the actual email and it contains one TO address (a
correct address, the only odd thing I noticed is that the TO address
is in all caps, and from looking at the DB/sphinx I know in some
places you were using "X" as a placeholder for periods.

I just migrated from 1.1.0 (since 2015) and set up a new server on
1.3.8 - so the issue would have been (at least with) 1.1.0.  My
upgrade method was a bit of a mix after reading all your documentation
on upgrading and server migrations (since I was doing both), as well
as Bitbucket issues. I know it's been a while, but I really wanted to
take the time to understand Piler before running an upgrade and
potentially screw something up.

I looked for a changelog with a fix like this, but I didn't see a
formal one, and looking through all the commits since 2015 was going
to take a LONG time so I figured I'd ask here. I can't see if I still
get new rcpt rows for that id, as I tossed my backups of prior DB
versions on the old server already, but I can certainly pay attention
on the new server.

select id, COUNT(*) from rcpt GROUP BY id HAVING COUNT(*) > 50 ORDER
BY COUNT(*) ASC; shows that there are definitely a couple other id's
with this issue, and likely many more, but there is a steep dropoff
from 3 million to 30k to 200-500 range per ID. Maybe you (or other
users) can use that query to check the recipient count for each stored
email for similar issues.



I just did, and found 10 such emails. Some of them had even ~150 
recipients.

After checking those 10 emails I found that they are all legitimate in
a sense that the email really consisted of that many recipient, and the
rcpt table entries were valid email addresses.

So fortunately the issue is not present in my personal archive.


I just wanted to make sure that my assumption on the rcpt table was
correct (e.g. it should only have the "to" recipients for each email
matching on metadata.id [1]), so that I can delete all those rows and


yes, you are right.


either manually insert the address found from pilerget or delete the
matching metadata row and just re-import the email to fix the issue.


In the latter case you'll get two hits for the same message from sphinx,
and one of them would be a ghost, because you deleted its corresponding
message.

Janos




On Sat, May 2, 2020 at 8:29 AM  wrote:


Hello Ryan,

this must be definitely a bug. What piler version do you use?
Try selecting the first 1000 rows for id=37, and check if the
recipients are actually all different.

Do you still get new rcpt rows for id=37?
I suspect that piler tries to keep processing the very same email.

Janos

On 2020-05-02 09:41, Ryan Blenis wrote:

Hello,

Just a question about the table. I took a look at the data, and

I've

got 23 million rows in there, however I kept seeing id 37 pop up.

I

queried for "SELECT COUNT(*) FROM rcpt WHERE id = 37" and got a

count

of 3,116,977.

I was under the impression (perhaps mistakenly) that the table was
used to store recipients of the email linked on metadata.id [1]

[1] - but

certainly no email had 3 million recipients. Can you let me know

what

I'm missing here? Thank you.




Links:
--
[1] http://metadata.id
[2] http://hotmail.com

Re: Rcpt Table Question

[sj]

Hello Ryan,

this must be definitely a bug. What piler version do you use?
Try selecting the first 1000 rows for id=37, and check if the
recipients are actually all different.

Do you still get new rcpt rows for id=37?
I suspect that piler tries to keep processing the very same email.

Janos


On 2020-05-02 09:41, Ryan Blenis wrote:

Hello,

Just a question about the table. I took a look at the data, and I've
got 23 million rows in there, however I kept seeing id 37 pop up. I
queried for "SELECT COUNT(*) FROM rcpt WHERE id = 37" and got a count
of 3,116,977.

I was under the impression (perhaps mistakenly) that the table was
used to store recipients of the email linked on metadata.id [1] - but
certainly no email had 3 million recipients. Can you let me know what
I'm missing here? Thank you.

Re: Piler 1.3.8 - config-site.php missing?

[sj]

Hello,

it's in /usr/local/etc/piler.

Janos


On 2020-04-21 19:42, BKH-Netz.de - Postfach wrote:

Hello,

i installed Piler 1.3.8 for testing on a new system like this:

wget https://bitbucket.org/jsuto/piler/downloads/piler-1.3.8.tar.gz
tar zxvf piler-1.3.8.tar.gz
cd piler-1.3.8/
./configure --localstatedir=/var --with-database=mysql 
--enable-memcached

make
make install
ldconfig
make postinstall

After doing so i noticed that there is no config-site.php in 
/var/piler/www/


How can i recreate this file?

Thanks
BKH

procedural documentation of piler

[sj]

Dear piler users,

there was a thread on GoBD compliance back in 2017. And the
conclusion was that all piler needs is a procedural documentation
describing what it does with an email.

Finally I've made it, thanks for Adrian to persuading me.

https://docs.google.com/document/d/1jqPu10WMgx9VLdsbOaSgiuNk9zgrHm2KfO7D_ci4jvc/

I'd like you to review it, and feel free to comment it, I've
setup the permissions allowing you to do so using google docs.
Or you may send your comments to the list or directly to me.

Thank you in advance,

Janos

Re: SSO Issues

[sj]

Hmm, it's odd. Anyway, I'm glad that you made it eventually. I'll check
what has changed since 1.37. Btw. the mail log is usually 
/var/log/mail.log

or /var/log/maillog.

Janos


On 2020-04-17 19:41, Robitaille, Cameron wrote:

I have it working now.  I downgraded the webui code from 1.38 to 1.37
and it magically started working.

Cameron Robitaille
Network Administrator
Waypoint Centre for Mental Health Care
Advancing Understanding. Improving Lives.

500 Church Street
Penetanguishene ON  L9M 1G3
(705) 549-3181 ext. 2834
crobitai...@waypointcentre.ca

This electronic transmission and accompanying attachments may contain
privileged or confidential information intended only for the use of
the individual or organization named above.  Any distribution, copy or
action taken in reliance of the contents of this communication by
anyone other than the intended recipient(s) is Strictly Prohibited. 
If you have received this communication in error, please notify the
sender at the above email address and delete this email.


-Original Message-
From: s...@acts.hu 
Sent: April 17, 2020 12:45 PM
To: Piler User 
Subject: Re: SSO Issues

CAUTION: This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender
and know the content is safe. If you are unsure, please delete the
message.

Hello,

look at the mail log file. Piler syslogs all events. For starters,
check config-site.php, and verify that you have enabled ldap
authentication.
Then check the ldap server, port, ldap helper user account settings.
Also check if there's any traffic from the archive to the ldap server.

Janos


On 2020-04-17 18:10, Robitaille, Cameron wrote:

Hi:

So I reinstalled everything according to documentation and again, I
can login as admin@local with no issue but I can't get any LDAP users
logged in even though the logs say they are authenticated.  Is there
any other logging I should be checking or turning on?

Thanks,

Cameron Robitaille
Network Administrator
Waypoint Centre for Mental Health Care Advancing Understanding.
Improving Lives.

500 Church Street
Penetanguishene ON  L9M 1G3
(705) 549-3181 ext. 2834
crobitai...@waypointcentre.ca

This electronic transmission and accompanying attachments may contain
privileged or confidential information intended only for the use of
the individual or organization named above.  Any distribution, copy or
action taken in reliance of the contents of this communication by
anyone other than the intended recipient(s) is Strictly Prohibited.
If you have received this communication in error, please notify the
sender at the above email address and delete this email.


-Original Message-
From: s...@acts.hu 
Sent: April 16, 2020 4:40 PM
To: Piler User 
Cc: Robitaille, Cameron 
Subject: Re: SSO Issues

CAUTION: This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender
and know the content is safe. If you are unsure, please delete the
message.

Hello Cameron,

based on the ntlm logs the authentication is successful, and apache
also says granted. Please check out the mail log as well as apache's
error log for additional clues.

Also check if you can login by using ldap authentication, ie. not
using sso.php but rather the usual login.php.

Janos


On 2020-04-16 21:49, Robitaille, Cameron wrote:


I am hoping there is an easy answer to this as I am about to pull my
hair out.  I am installing to a new server in the hopes of migrating
from an existing install.

I have everything installed and I can log in as the admin@local user.
I configured the LDAP auth with the SSO configuration and I can run
and get good results from manually running ntlm_auth with a valid
user.  I copied the working config from the other server into my
config-site.php.  When I try to SSO it just spins although the log
says:

[Thu Apr 16 15:40:47.512194 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(1041): [client 10.10.95.2:31200] doing ntlm
auth dance [Thu Apr 16 15:40:47.512209 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(484): [client 10.10.95.2:31200] Using
existing auth helper 4591 [Thu Apr 16 15:40:47.512238 2020] [:debug]
[pid 4441]
mod_auth_ntlm_winbind.c(703): [client 10.10.95.2:31200] parsing reply
from helper to KK
TlRMTVNTUAADGAAYAIBcAVwBmABYFgAWAFgSABIAbgD0AQAABYKIogoAukcPBEnAms1Y9ke05O4HIhc6wmMAcgBvAGIAaQB0AGEAaQBsAGwAZQBEADQANwA2ADYAMwA1ADAAVgB8kPWCQbl+hlzCxJzjB1b2AQEAAABUXDbtJhTWAV0K8qKcAAGzAAIAFABNAEgAQwBQAC4ATwBOAC4AQwBBAAEAEABNAEEASQBMAEEAUgBDAEgABAAiAHcAYQB5AHAAbwBpAG4AdABjAGUAbgB0AHIAZQAuAGMAYQADADQAbQBhAGkAbABhAHIAYwBoAC4AdwBhAHkAcABvAGkAbgB0AGMAZQBuAHQAcgBlAC4AYwBhAAcACABUXDbtJhTWAQYABAACCAAwADAAAQAgAAD2cE88eSCt3N1kv//zYg08RWAN2pKfP+fpHHmqo/q6ywoAEACb+cMBW3SC6HqWBNugfGr2CQA+AEgAVABUAFAALwBtAGEAaQBsAGEAcgBjAGgALgB3AGEAeQBwAG8AaQBuAHQAYwBlAG4AdAByAGUALgBjAGE=\n
[Thu 

Re: SSO Issues

[sj]

Hello,

look at the mail log file. Piler syslogs all events. For starters,
check config-site.php, and verify that you have enabled ldap 
authentication.

Then check the ldap server, port, ldap helper user account settings.
Also check if there's any traffic from the archive to the ldap server.

Janos


On 2020-04-17 18:10, Robitaille, Cameron wrote:

Hi:

So I reinstalled everything according to documentation and again, I
can login as admin@local with no issue but I can't get any LDAP users
logged in even though the logs say they are authenticated.  Is there
any other logging I should be checking or turning on?

Thanks,

Cameron Robitaille
Network Administrator
Waypoint Centre for Mental Health Care
Advancing Understanding. Improving Lives.

500 Church Street
Penetanguishene ON  L9M 1G3
(705) 549-3181 ext. 2834
crobitai...@waypointcentre.ca

This electronic transmission and accompanying attachments may contain
privileged or confidential information intended only for the use of
the individual or organization named above.  Any distribution, copy or
action taken in reliance of the contents of this communication by
anyone other than the intended recipient(s) is Strictly Prohibited. 
If you have received this communication in error, please notify the
sender at the above email address and delete this email.


-Original Message-
From: s...@acts.hu 
Sent: April 16, 2020 4:40 PM
To: Piler User 
Cc: Robitaille, Cameron 
Subject: Re: SSO Issues

CAUTION: This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender
and know the content is safe. If you are unsure, please delete the
message.

Hello Cameron,

based on the ntlm logs the authentication is successful, and apache
also says granted. Please check out the mail log as well as apache's
error log for additional clues.

Also check if you can login by using ldap authentication, ie. not
using sso.php but rather the usual login.php.

Janos


On 2020-04-16 21:49, Robitaille, Cameron wrote:


I am hoping there is an easy answer to this as I am about to pull my
hair out.  I am installing to a new server in the hopes of migrating
from an existing install.

I have everything installed and I can log in as the admin@local user.
I configured the LDAP auth with the SSO configuration and I can run
and get good results from manually running ntlm_auth with a valid
user.  I copied the working config from the other server into my
config-site.php.  When I try to SSO it just spins although the log
says:

[Thu Apr 16 15:40:47.512194 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(1041): [client 10.10.95.2:31200] doing ntlm
auth dance [Thu Apr 16 15:40:47.512209 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(484): [client 10.10.95.2:31200] Using existing
auth helper 4591 [Thu Apr 16 15:40:47.512238 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(703): [client 10.10.95.2:31200] parsing reply
from helper to KK
TlRMTVNTUAADGAAYAIBcAVwBmABYFgAWAFgSABIAbgD0AQAABYKIogoAukcPBEnAms1Y9ke05O4HIhc6wmMAcgBvAGIAaQB0AGEAaQBsAGwAZQBEADQANwA2ADYAMwA1ADAAVgB8kPWCQbl+hlzCxJzjB1b2AQEAAABUXDbtJhTWAV0K8qKcAAGzAAIAFABNAEgAQwBQAC4ATwBOAC4AQwBBAAEAEABNAEEASQBMAEEAUgBDAEgABAAiAHcAYQB5AHAAbwBpAG4AdABjAGUAbgB0AHIAZQAuAGMAYQADADQAbQBhAGkAbABhAHIAYwBoAC4AdwBhAHkAcABvAGkAbgB0AGMAZQBuAHQAcgBlAC4AYwBhAAcACABUXDbtJhTWAQYABAACCAAwADAAAQAgAAD2cE88eSCt3N1kv//zYg08RWAN2pKfP+fpHHmqo/q6ywoAEACb+cMBW3SC6HqWBNugfGr2CQA+AEgAVABUAFAALwBtAGEAaQBsAGEAcgBjAGgALgB3AGEAeQBwAG8AaQBuAHQAYwBlAG4AdAByAGUALgBjAGE=\n
[Thu Apr 16 15:40:47.519444 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(741): [client 10.10.95.2:31200] got response:
AF MHCP.ON.CA\\user
[Thu Apr 16 15:40:47.519499 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(787): [client 10.10.95.2:31200] authenticated
MHCP.ON.CA\\user [Thu Apr 16 15:40:47.519522 2020] [authz_core:debug]
[pid 4441]
mod_authz_core.c(809): [client 10.10.95.2:31200] AH01626:
authorization result of Require valid-user : granted [Thu Apr 16
15:40:47.519535 2020] [authz_core:debug] [pid 4441]
mod_authz_core.c(809): [client 10.10.95.2:31200] AH01626:
authorization result of : granted

I would read this that it was successful but the page just spins and
nothing ever happens, the search never comes up.

I am using version 1.38.  Thanks,

Cameron Robitaille
Network Administrator
Waypoint Centre for Mental Health Care Advancing Understanding.
Improving Lives.

500 Church Street
Penetanguishene ON  L9M 1G3
(705) 549-3181 ext. 2834
crobitai...@waypointcentre.ca

This electronic transmission and accompanying attachments may contain
privileged or confidential information intended only for the use of
the individual or organization named above.  Any distribution, copy or
action taken in reliance of the contents of this communication by
anyone other than the intended recipient(s) is Strictly Prohibited.
If you have received 

Re: SSO Issues

[sj]

Hello Cameron,

based on the ntlm logs the authentication is successful, and apache
also says granted. Please check out the mail log as well as apache's
error log for additional clues.

Also check if you can login by using ldap authentication, ie. not using 
sso.php

but rather the usual login.php.

Janos


On 2020-04-16 21:49, Robitaille, Cameron wrote:


I am hoping there is an easy answer to this as I am about to pull my
hair out.  I am installing to a new server in the hopes of migrating
from an existing install.

I have everything installed and I can log in as the admin@local user.
I configured the LDAP auth with the SSO configuration and I can run
and get good results from manually running ntlm_auth with a valid
user.  I copied the working config from the other server into my
config-site.php.  When I try to SSO it just spins although the log
says:

[Thu Apr 16 15:40:47.512194 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(1041): [client 10.10.95.2:31200] doing ntlm
auth dance
[Thu Apr 16 15:40:47.512209 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(484): [client 10.10.95.2:31200] Using existing
auth helper 4591
[Thu Apr 16 15:40:47.512238 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(703): [client 10.10.95.2:31200] parsing reply
from helper to KK
TlRMTVNTUAADGAAYAIBcAVwBmABYFgAWAFgSABIAbgD0AQAABYKIogoAukcPBEnAms1Y9ke05O4HIhc6wmMAcgBvAGIAaQB0AGEAaQBsAGwAZQBEADQANwA2ADYAMwA1ADAAVgB8kPWCQbl+hlzCxJzjB1b2AQEAAABUXDbtJhTWAV0K8qKcAAGzAAIAFABNAEgAQwBQAC4ATwBOAC4AQwBBAAEAEABNAEEASQBMAEEAUgBDAEgABAAiAHcAYQB5AHAAbwBpAG4AdABjAGUAbgB0AHIAZQAuAGMAYQADADQAbQBhAGkAbABhAHIAYwBoAC4AdwBhAHkAcABvAGkAbgB0AGMAZQBuAHQAcgBlAC4AYwBhAAcACABUXDbtJhTWAQYABAACCAAwADAAAQAgAAD2cE88eSCt3N1kv//zYg08RWAN2pKfP+fpHHmqo/q6ywoAEACb+cMBW3SC6HqWBNugfGr2CQA+AEgAVABUAFAALwBtAGEAaQBsAGEAcgBjAGgALgB3AGEAeQBwAG8AaQBuAHQAYwBlAG4AdAByAGUALgBjAGE=\n
[Thu Apr 16 15:40:47.519444 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(741): [client 10.10.95.2:31200] got response:
AF MHCP.ON.CA\\user
[Thu Apr 16 15:40:47.519499 2020] [:debug] [pid 4441]
mod_auth_ntlm_winbind.c(787): [client 10.10.95.2:31200] authenticated
MHCP.ON.CA\\user
[Thu Apr 16 15:40:47.519522 2020] [authz_core:debug] [pid 4441]
mod_authz_core.c(809): [client 10.10.95.2:31200] AH01626:
authorization result of Require valid-user : granted
[Thu Apr 16 15:40:47.519535 2020] [authz_core:debug] [pid 4441]
mod_authz_core.c(809): [client 10.10.95.2:31200] AH01626:
authorization result of : granted

I would read this that it was successful but the page just spins and
nothing ever happens, the search never comes up.

I am using version 1.38.  Thanks,

Cameron Robitaille
Network Administrator
Waypoint Centre for Mental Health Care
Advancing Understanding. Improving Lives.

500 Church Street
Penetanguishene ON  L9M 1G3
(705) 549-3181 ext. 2834
crobitai...@waypointcentre.ca

This electronic transmission and accompanying attachments may contain
privileged or confidential information intended only for the use of
the individual or organization named above.  Any distribution, copy or
action taken in reliance of the contents of this communication by
anyone other than the intended recipient(s) is Strictly Prohibited. 
If you have received this communication in error, please notify the
sender at the above email address and delete this email.

Re: Piler - Backup and Restore

[sj]

Hello,

your scripts look fine by looking. Do the following to troubleshoot
the issue:

check the permissions on /var/piler, piler should be able read it,
then /var/piler/sphinx dir and its contents must be read-writable
for piler.

When you start searchd, verify that searchd runs as user piler, and
can read all the index files. It prints the indices it just read.

Then try connecting to searchd: mysql -h 127.0.0.1 -P9306, and
run "select * from main1;". It should print 20 hits or so.

Janos



On 2020-04-16 03:07, BKH-Netz.de - Postfach wrote:

Hi Janos,

i’m trying to gat a working backup / restore script for my piler 
installations.

But my efforts are more or less without a success …
Perhaps you can give me a slight hint?

My backup script looks like this:

- - - - - - -
systemctl stop rc.piler.service
systemctl stop rc.searchd.service

tar -cpzf SOMEVZ/piler.tar.gz -C /usr/local/etc/piler .
tar -cpzf SOMEVZ//pilerstore.tar.gz -C /var/piler/store/00 .
tar -cpzf SOMEVZ/sphinx.tar.gz /var/piler/sphinx/main[1-4]*

mysqldump --single-transaction -h localhost -piler -p'mysecretpw'
piler > SOMEVZ/pilerdb.sql

systemctl start rc.piler.service
systemctl start rc.searchd.service
- - - - - - -

My restore script looks like this:

- - - - - - -
systemctl stop rc.piler.service
systemctl stop rc.searchd.service

rm -r /usr/local/etc/piler
rm -r /var/piler/store/00
rm /var/piler/sphinx/main[1-4]*

mkdir -p /usr/local/etc/piler
mkdir -p /var/piler/store/00
tar -xpzf SOMEVZ/piler.tar.gz -C /usr/local/etc/piler
tar -xpzf SOMEVZ//pilerstore.tar.gz -C /var/piler/store/00
tar -xpzf SOMEVZ/sphinx.tar.gz -C /

mysql -h localhost -upiler -p'mysecretpw' piler < SOMEVZ/pilerdb.sql

systemctl start rc.piler.service
systemctl start rc.searchd.service
- - - - - - -

All imported mails (figures) are show correctly inside the admin 
interface.

Searching for mail in the auditor interface shows no emails at all.
I guess this is a problem with sphinx database but i cant figure out …

Can you give me a little help on this please?

HW

Re: S3 Object Storage

[sj]

Hello Koby,

On 2020-04-06 07:07, Koby Peleg Hen wrote:

Hello all ,

I am currently on this ,

Please Janos , make your decision.

Since I have looking at your project for very long time ,

And my client begin to ask for.

It is very simple to implenent , because there is simple convertor
from regular file system to S3.

If you decided  not to , please let us all know.


the answer is no for now. I've googled the topic, and found s3fs 
(perhaps

you referred to the same project).

Found a few articles that might come handy:

https://cloud.netapp.com/blog/amazon-s3-as-a-file-system
https://medium.com/swlh/using-s3-just-like-a-local-file-system-in-python-497737783f11

I encourage you and other piler users as well to test it. Perhaps it's
a good enough solution, and works with piler properly. If so, then I'll
update the docs with your findings.



In this case I will need to implement it by my self by the near
future.

I have a lot of respect to your project and your time and the willing
to get pay for it.

But if some one whould like to pay , is for the support not for
feature.

Please take a look on the proxmox project.


I just did. And looked at nginx plus as well. Some projects monetize 
support,

others the features as well.



By my self , and my clients , I do not mean to take off without S3.


I'd say give a shot for s3fs, and see how it works. I'm not against it 
(or

any usable solution) by any means.

Janos SUTO

Re: S3 Object Storage

[sj]

Hello Ryan,

On 2020-04-06 00:45, Ryan Blenis wrote:


Thank you as always for your quick and in-depth response. I certainly
understand the delicate balance between FOSS and enterprise paywalls
and feature-sets, and I thank you for even having an open source
option to begin with! I'm just planning a migration to a new server
(for when my current OS is EOL) and wanted to move the hosting to one
with better support that doesn't over-provision the hosts resources as
I've seen on the current host, but HDD space isn't anywhere near as
inexpensive as it is at the current company. I was just thinking I
could have the best of both worlds with S3 storage integrated. Many of
the FOSS/enterprise paywalls in other projects are based on LDAP
integration, multi-tenancy, pre-packages binaries, support levels,
etc. that many business need, whereas object storage vs block storage
felt like more of a separate paywall in terms of cloud providers, so I
just figured I'd ask if it was coming to the FOSS version. Also, I


it's ok to ask, and I always appreciate the feedback from piler users.
Btw. can you share the names of these 2 providers? I'm curious about
their pricing. And perhaps the piler community may suggest some better
alternatives both in terms of hw performance and storage pricing.



like to be able to modify the code if a customization can be made,
which I assume would not be the case with enterprise, correct?


Not sure if you fix the C source code before compiling. If not, then
it's probably not a big deal that you can't modify the enterprise 
edition

binaries. However, the gui part is still the usual php based stuff,
it's not obfuscated, so you can fix it if you really want to do that.
It's your archive after all. Custom css is supported out of the box.


Also, if I try the enterprise on the new host, is there a path to
downgrade to the FOSS version? Or would starting back from scratch be
the only option?


Unfortunately no. The enterprise edition handles attachments in a 
different
way: it doesn't need the attachment table to retrieve the message with 
its

attachments. However, it's incompatible with the open source version.

So if you decide later that it's not what you want then you should 
export

all emails, install the open source edition, and import those emails.
Though, I'm confident that you'll like it, it's not inferior to the
open source edition.


Janos SUTO


On Sun, Apr 5, 2020 at 7:16 AM  wrote:


Hello Ryan,

On 2020-04-05 10:55, Ryan Blenis wrote:


I see the enterprise version you offer it says one of the features
piler has is S3 object storage, which looks like it was originally
requested on the mailing list in 2018:
https://www.mail-archive.com/piler-user@list.acts.hu/msg01335.html

Does this mean that it is fully implemented now? And if so, will

we

yes, it's fully implemented by now.


ever see this in the open source version? Given the lower prices

on S3

compatible storage versus block storage and the unchanging nature

of

the files, this would be great to have as an option that can

reduce

hosting costs from what is now being used for [comparatively]
expensive block storage.


well, I've been thinking about it for a while, and still not
decided,
it's a difficult business decision, so let me share my current
position
on the matter.

TL;DR: not in the near future.

I'm really proud that the open source edition has gained a pretty
nice
traction, and I decided to somehow monetize its popularity.

So I've forked piler and built an enterprise edition along with some

commercial
services. To make the enterprise edition successful, I must offer
some
features
that are worth to pay for that is not available for free in the open

source edition.

I believe that the S3 storage support you asked for is such an
attractive feature,
so I'll keep it for the enterprise edition only, not sure how long.

However, I'm well aware of that the S3 costs are significantly lower

than block
storage costs at cloud hosting companies. I encourage you to try the

enterprise
edition, and see if it works for you. If so, and you like it, then
perhaps we
could figure out something viable to make the transition from open
source to
enterprise. I'd like to emphasize that I don't want to force you in
this
direction,
and it's totally ok to decline.

Let me know what you (and other piler users as well) think about the

matter.

Janos

Re: S3 Object Storage

[sj]

Hello Ryan,

On 2020-04-05 10:55, Ryan Blenis wrote:


I see the enterprise version you offer it says one of the features
piler has is S3 object storage, which looks like it was originally
requested on the mailing list in 2018:
https://www.mail-archive.com/piler-user@list.acts.hu/msg01335.html

Does this mean that it is fully implemented now? And if so, will we


yes, it's fully implemented by now.



ever see this in the open source version? Given the lower prices on S3
compatible storage versus block storage and the unchanging nature of
the files, this would be great to have as an option that can reduce
hosting costs from what is now being used for [comparatively]
expensive block storage.



well, I've been thinking about it for a while, and still not decided,
it's a difficult business decision, so let me share my current position
on the matter.

TL;DR: not in the near future.

I'm really proud that the open source edition has gained a pretty nice
traction, and I decided to somehow monetize its popularity.

So I've forked piler and built an enterprise edition along with some 
commercial
services. To make the enterprise edition successful, I must offer some 
features
that are worth to pay for that is not available for free in the open 
source edition.


I believe that the S3 storage support you asked for is such an 
attractive feature,

so I'll keep it for the enterprise edition only, not sure how long.

However, I'm well aware of that the S3 costs are significantly lower 
than block
storage costs at cloud hosting companies. I encourage you to try the 
enterprise
edition, and see if it works for you. If so, and you like it, then 
perhaps we
could figure out something viable to make the transition from open 
source to
enterprise. I'd like to emphasize that I don't want to force you in this 
direction,

and it's totally ok to decline.

Let me know what you (and other piler users as well) think about the 
matter.


Janos

Re: Upgrade from 1.34 to 1.37

[sj]

Hello Cameron,

well, for starters check out 
http://www.mailpiler.org/wiki/current:upgrade

Basically you get the tarball, run configure with the same parameters as
before, run make clean all, make install. And update the gui as well.
Then restart piler daemons, and you should be fine.

Janos

On 2020-02-26 16:07, Robitaille, Cameron wrote:

Hi:

I realize this is a minor update but I had a quick question about it.
Do I simply follow the install instructions again to get the upgrade
complete or is there some other things I need to do?

Thanks,

Cameron Robitaille

Network Administrator

Waypoint Centre for Mental Health Care

Advancing Understanding. Improving Lives.

500 Church Street

Penetanguishene ON  L9M 1G3

(705) 549-3181 ext. 2834

crobitai...@waypointcentre.ca

This electronic transmission and accompanying attachments may contain
privileged or confidential information intended only for the use of
the individual or organization named above.  Any distribution, copy or
action taken in reliance of the contents of this communication by
anyone other than the intended recipient(s) is Strictly Prohibited.
If you have received this communication in error, please notify the
sender at the above email address and delete this email.

Re: Wrong dates

[sj]

The date header is malformed: Date: Thu, 23 Jan 2020 20:11:30 UT

Notice the missing "C" letter. Once I appended it, the recognized
timestamp was just fine.

Janos

On 2020-02-04 12:55, Marcelo Machado wrote:

Thank you Janos. Below is the header:

Re: groups based on LDAP attribute

[sj]

Hello Martin,

On 2020-01-07 17:26, Martin Nadvornik wrote:


although my answer is a bit late, I still want to thank you for your
explanation and example. It worked as expected for our use case. I
already knew about the custom authentication function but I never
thought of performing an ldap bind there to get the required addresses
and add them to the session data.


no worries, you are welcome :-)

Janos




Am 22.11.2019 um 21:46 schrieb s...@acts.hu:



Hello Martin,

my first idea was the group feature until I saw you had ruled it out.

Then the custom email query function is to the rescue, see
http://www.mailpiler.org/wiki/current:custom-authentication for more.

So let's say you have an ldap attribute called 'aaa' with value 'bbb'
having the required email addresses, and the manager is
mana...@yourdomain.com.

Then put a similar function to config-site.php:

$config['CUSTOM_EMAIL_QUERY_FUNCTION'] = 'my_custom_func';

function my_custom_func($username = '') {
   $session = Registry::get('session');
   $data = $session->get("auth_data");

   $extra_emails = [];

   if($data['email'] == 'mana...@yourdomain.com') {
  $ldap = new LDAP($ldap_host, $ldap_port, $ldap_helper_dn,
$ldap_helper_password);

  if($ldap->is_bind_ok()) {
 $query = $ldap->query($ldap_base_dn, "aaa=bbb");
 $extra_emails = $query->rows;
  }
   }

   $data['emails'] = array_merge($data['emails'] , $extra_emails);

   $session->set("auth_data", $data);
}

Note that you may have to fine tune the ldap query, need to do some
polishing on the $query->rows
part. Also be sure to check out checkLoginAgainstLDAP_real() function
in model/user/auth.php
to see how it works.


Janos

On 2019-11-22 20:30, Martin Nadvornik wrote:

Hello fellow piler users,

I am currently trying to find a way to give managers the ability to
search e-mails of employees who are not anymore with the company. I 
know

that in general you can give the owner of one address the ability to
access other addresses by using an alias or distribution list. 
However
in our case the e-mail accounts of prior employees don't exist 
anymore
and their e-mail addresses should not be reachable. Therore using 
alias

or distribution lists are not an option.
It would be possible of course to create groups directly in piler but
maintaining such groups by hand is not really efficient. So I am 
looking

for a way to use the group feature using LDAP.
The easiest method for us would be if there was a config value that 
just

tells piler which LDAP attribute to use and just allows the logged in
user access to every addresses listed in the attribute. But this 
needs

some development effort as far as I can tell.

Is there anyone who has done this before? Any recommendations to 
solve

this are welcome.

Thanks,
Martin




--
Martin Nadvornik
IT Mitarbeiter

Diakonie Flüchtlingsdienst gem. GmbH
Steinergasse 3/12
1170 Wien, Austria
Tel: +43 (0) 1/ 402 67 54 - 1337
Fax: +43 (0) 1/ 402 67 54 - 16
Mobil: 0664/88350692
http://www.diakonie.at/fluechtlingsdienst

Verantwortung kann man nicht abschieben!

Re: groups based on LDAP attribute

[sj]

Hello Martin,

my first idea was the group feature until I saw you had ruled it out.

Then the custom email query function is to the rescue, see
http://www.mailpiler.org/wiki/current:custom-authentication for more.

So let's say you have an ldap attribute called 'aaa' with value 'bbb'
having the required email addresses, and the manager is 
mana...@yourdomain.com.


Then put a similar function to config-site.php:

$config['CUSTOM_EMAIL_QUERY_FUNCTION'] = 'my_custom_func';

function my_custom_func($username = '') {
   $session = Registry::get('session');
   $data = $session->get("auth_data");

   $extra_emails = [];

   if($data['email'] == 'mana...@yourdomain.com') {
  $ldap = new LDAP($ldap_host, $ldap_port, $ldap_helper_dn, 
$ldap_helper_password);


  if($ldap->is_bind_ok()) {
 $query = $ldap->query($ldap_base_dn, "aaa=bbb");
 $extra_emails = $query->rows;
  }
   }

   $data['emails'] = array_merge($data['emails'] , $extra_emails);

   $session->set("auth_data", $data);
}

Note that you may have to fine tune the ldap query, need to do some 
polishing on the $query->rows
part. Also be sure to check out checkLoginAgainstLDAP_real() function in 
model/user/auth.php

to see how it works.


Janos

On 2019-11-22 20:30, Martin Nadvornik wrote:

Hello fellow piler users,

I am currently trying to find a way to give managers the ability to
search e-mails of employees who are not anymore with the company. I 
know

that in general you can give the owner of one address the ability to
access other addresses by using an alias or distribution list. However
in our case the e-mail accounts of prior employees don't exist anymore
and their e-mail addresses should not be reachable. Therore using alias
or distribution lists are not an option.
It would be possible of course to create groups directly in piler but
maintaining such groups by hand is not really efficient. So I am 
looking

for a way to use the group feature using LDAP.
The easiest method for us would be if there was a config value that 
just

tells piler which LDAP attribute to use and just allows the logged in
user access to every addresses listed in the attribute. But this needs
some development effort as far as I can tell.

Is there anyone who has done this before? Any recommendations to solve
this are welcome.

Thanks,
Martin

Comparison of piler vs google vault/office 365

[sj]

Dear piler-users,

I got a request recently to compare piler with office 365 and google 
vault.


Since both google and o365 offer a cloud based solution with some minor 
or perhaps
not so minor differences between them, I put both them to the blue 
corner against
piler in the red corner. Please note that this is my somewhat subjective 
opinion,
and as usual, your mileage may wary. Also feel free to add your own 
thoughts to

the matter.

If you already have your messaging at Google or o365, then it's a 
trivial choice
to use their email archiving solutions. When I heard that o365 had been 
launched
and offering archiving as well, I saw lots of commercial archiving 
vendors writing
blog posts justifying their values, and showing (or at least trying) why 
their
products are far superior to both G vault and O365. Though their 
arguments were

valid, I noticed some fear of losing some of their market share as well.

I believe that each solutions have pros and cons, and your task is to 
find the

best solution matching your needs.


Advantages of G Suite/O365
---

- saas solution, ready to use without much hassle or fuss
- easy to calculate monthly/yearly costs (opex only)
- no need to buy a server
- no need to hire a Linux ninja to run your archive
- includes the mail hosting*
- monthly fee includes support**
- strong companies behind them

Drawbacks of G Suite/O365
-

- extra cost*** for the archiving feature
  i) Google Business costs +$6/mo/user, $7200/100 user/year (compared to 
the Basic plan)
  ii) Office 365 E3 costs +$12/mo/user, $14000/100 user/year (compared 
to E1 plan)


- suspend users that have left the company (rather than delete them) to 
retain their data
- only available in the cloud. Some organizations must keep their data 
on premise

- restore message to the inbox is quite complicated for G vault
- lack of the ability to import legacy emails
- difficulties keeping the archive when migrating to another mail 
provider



*: No doubt, running your mail hosting for yourself requires pretty 
decent Linux
and other tech skills, eg. setup some sort of spam and other malware 
protection,
not to mention the backup, high availability, and other ninja level 
stuff


**: I haven't used their support, so I cannot confirm their quality, 
responsiveness, etc


***: To be fair I have to acknowledge that the bigger/more expensive 
plans usually
offer more for the extra besides archiving, eg. more services or more 
storage, etc.


: I couldn't find it how to import legacy emails to G vault in the 
docs. O365 supports
importing from pst files, but it still sucks an is painful (though it's 
true for any migration

involving TBs of data, so it's partly not o365's fault).


Advantages of piler
---

- open source
- no extra cost
- lots of handy features
- easy to import legacy emails
- exports to standard format -> easy to leave piler for another product
- you can keep your archive even if you migrate to another mail provider
- good fit if you want to keep your data on premise
- you can customize it
- nice support* :-)

Drawbacks of piler
--

- you need a host and it costs (capex) **
- it requires a pretty skilled Linux ninja to setup and run it for 
yourself
  (70+% of the issues stem from missing some skills, or simply 
understanding how piler works)

- despite I do pushups, I'm not so strong like G or o365
- you have to backup for yourself

*: even though I have some stuck issues in bitbucket

**: you can still deploy piler to the cloud, eg. aws, azure, ... 
which is still opex




Sources:
- https://blog.microfocus.com/top-10-google-vault-archiving-drawbacks/
- 
https://spanning.com/blog/google-vault-and-spanning-backup-two-great-products-two-different-purposes/
- 
https://www.damsoncloud.com/2017/03/third-party-backups-essential-even-use-google-vault/
- 
https://www.quora.com/If-youre-using-Google-Apps-what-email-archival-system-if-any-do-you-use

- https://www.intradyn.com/email-archiving-alternative-google-vault/
- 
https://www.microsoft.com/en-us/microsoft-365/blog/2015/05/07/making-email-archive-migration-easier-with-the-office-365-import-service/



So that's it, let me know what you think.

Janos


PS: before you ask: since I was working for a service provider in the 
past, piler is an obvious choice for me

Re: Authentication against multiple IMAP servers

[sj]

Hello,

I've fixed the typo in the docs. I suggest you to also set IMAP_PORT and 
IMAP_SSL parameters.

You may run ngrep to figure out what's going on the wire.

Janos

On 2019-10-31 13:15, hw wrote:

Hello Janos,

thanks so far :-)

I found the typo and it has to be like this:

$config['CUSTOM_PRE_AUTH_FUNCTION'] = 'my_imap';

function my_imap($username = '') {
   global $session;

   if(strstr($username, "@domain1.com")) {
  define('IMAP_SERVER', 'imap.domain1.com');
   } else if(strstr($username, "@domain2.com")) {
  define('IMAP_SERVER', 'imap.domain2.com');
   } else {
  define('IMAP_SERVER', 'imap.somedomain.com');
   }
}

Now the login page works :-)

I changed values to my own domain but i get this persistent error:

019/10/31 13:08:05 [error] 27070#27070: *9 FastCGI sent in stderr:
"PHP message: PHP Warning:  stream_socket_client(): unable to connect
to ssl://mail.yourdomain.com:993 (Connection timed out) in
/var/piler/www/Zend/Mail/Protocol/Imap.php on line 108" while reading
response header from upstream, client: XXX.XXX.XXX.XXX, server:
my.example-archive.de, request: "POST /login.php HTTP/2.0", upstream:
"fastcgi://unix:/run/php/php7.3-fpm.sock:", host:
"my.example-archive.de"

It seems server variables are not set correctly.

BKH

Re: Authentication against multiple IMAP servers

[sj]

Hello,

I've just updated the custom authentication docs at
http://www.mailpiler.org/wiki/current:custom-authentication

Please check out the CUSTOM_PRE_AUTH_FUNCTION feature at the end.

Janos

On 2019-10-29 10:12, BKH - Post wrote:

Hi @all,

is there a possibility to establish an authentication against multiple
IMAP servers?
Having here a scenario with two to three internal IMAP servers in
different workgroups i don't want to setup three mailpiler instances.
In fact that setting setting up such system is not a problem i want to
reduce administrative tasks.

Thanks for your advice.

Regards

BKH

Re: Deletion / Purging of maiil - mail are shown in searchlist

[sj]

Hello,

I've just redeployed my dev environment. This is how it works (and it 
works):


I selected 3 emails, then as an auditor I removed them. On the gui they 
were

greyed out immediately.

Then I ran pilerpurge.py, and noticed in the mail logs:

Aug 28 18:39:00 780405577d9b pilerpurge.py[957]: Purged 3 messages, 4 
files, 22098/11248 bytes


However, even though the messages were removed from disk, they were 
still present
in the sphinx database. So I could see them in the upper pane. When I 
clicked either
of them, then lower pane was basically empty (because the messages were 
gone).


Now the caveat: after running the delta indexer, nothing has changed on 
the gui.
Then after running the daily main indexer, still nothing. The first 
delta indexer
running right after the main indexer did the trick, because the sphinx 
killlist

(k-list) is tied to the main index.

Janos


On 2019-08-28 10:08, BKH - Post wrote:


two days ago i deleted some mails for testing reasons with 
pilerpurge.py

as it is shown here:
http://www.mailpiler.org/wiki/current:purging-aged-emails?s[]=%2Apurge%2A

Looking into searchlist with auditor privileges i can see those delted 
/

purged mails even today.
So i don't know what went wrong here because syslog shows no entries.
Documentation mentiones a killlist feature which i assume to be inside
delta indexer.
Delta indexer runs every 10 minutes in my testing environment.

I'm unsing sphinxsearch 3.11 and piler 1.3.5.

How to "update" sphinxsearch?

Thanks for your help

BKH

Re: Cronjobs - change indexing interval

[sj]

Hello,

On 2019-08-25 18:29, BKH wrote:


unfortunately all older emails were gone in WebGUI.
After checking sphinx index i found them in spinx index.
I dont know how to get then back to WebGUI an so i tried to reindex.
But this did not work for me.

In fact that this is not a big problem - yes, i made some backups - i'm
going to set up a new testing environment.
But how to get back emails when they are gone in WebGUI but are found 
in

sphinx index?



it appears that the index data defined for sphinx v3 is not compatible 
with

v2 (and vice versa). So do the following:

- stop searchd
- rm -f /var/piler/sphinx/*
- su -c "indexer --all --config /usr/local/etc/piler/sphinx.conf" piler
- verify that you have files owned by piler in /var/piler/sphinx/
- startd searchd
- reindex all emails, ie. su - piler; cd /tmp; reindex -f 1 -t 100; 
/usr/local/libexec/piler/indexer.delta.sh


After this you should see some emails in the gui. If so, run reindex -a 
(as user piler)

to reindex all emails.

Janos

mobile theme removal

[sj]

Dear piler users,

I'm planning to remove the mobile theme unless some of you are
using it actively. In this case drop me an email that I shouldn't.
The deadline is 1 week from now.

The mobile theme is my clumsy attempt to create a mobile friendly
theme, however the default theme still looks much better, and
it should be ok on mobile devices as well.

Janos

Re: Cronjobs - change indexing interval

[sj]

Hello,

On 2019-08-24 16:11, BKH wrote:


i put

define('SPHINX_VERSION', 311);


if a similar line wasn't there already, then your sphinx.conf version is
not ready for 3.1.1. Grab this one:
https://bitbucket.org/jsuto/piler/raw/c84bea95061820135cc1edcc8beb88ccb0d0f053/etc/sphinx.conf.in
and save it as sphinx.conf (note that @LOCALSTATEDIR@ must be fixed at 
the beginning).


Set 311 for the sphinx version, then check if searchd is happy with it:
(it will print any issues it encounters)

searchd -c /usr/local/etc/piler/sphinx.conf

if so, then kill it, and restart it with the rc script or systemctl.

Janos

Re: Purging aged / deleted mails

[sj]

Hello,

did you fix the sphinx version number in sphinx.conf?
For sphinx 3.1.1 it should be

define('SPHINX_VERSION', 311);

Janos

On 2019-08-24 15:34, BKH wrote:

Hello,

i made some progress in building a mail archiv with the help of piler.

Now i tried to purge some aged / deleted (i know, it does not comply
with law, but for testing reasons ...).

I used "pilerpurge.py" for doing this in the following way

a) /usr/local/libexec/piler/pilerpurge.py -d -c
/usr/local/etc/piler/piler.conf (dry-run)

/var/piler/store/00/5d5/45/bc/40005d5fc54e05e6008c009d46a045bc.m
/var/piler/store/00/5d5/cc/47/40005d5fc6152548bf14007cd1a2cc47.m
/var/piler/store/00/5d5/62/78/40005d5fc72926c174940087c1266278.m
/var/piler/store/00/5d5/34/b2/40005d5fc9d60b001bd400d981fc34b2.m
/var/piler/store/00/5d5/6c/21/40005d5fcf3b1b696afc00ae920f6c21.m
/var/piler/store/00/5d5/9c/28/40005d5fcf3b2b5b441c00ef888c9c28.m
/var/piler/store/00/5d5/ab/82/40005d5fcf3b39c0cc3400849050ab82.m
/var/piler/store/00/5d5/a4/2b/40005d5fcf3c0b70005c004e07c3a42b.m
/var/piler/store/00/5d5/69/35/40005d5fcf5606efa08c000e2d176935.m
/var/piler/store/00/5d5/6c/7a/40005d5fd0671b52702c0046ff4e6c7a.m
/var/piler/store/00/5d5/41/ff/40005d5fd06729cfa2dc00077df841ff.m

b) /usr/local/libexec/piler/pilerpurge.py -c 
/usr/local/etc/piler/piler.conf


I waited abount ~30 minutes for for delta indexer, which indexed 3 
times

(i changed index interval in crontab), but all deleted mails are shown.

Looking into syslog i manually started die delta indexer and got some
messages i don't understand:

/usr/local/libexec/piler/indexer.delta.sh

WARNING: key 'docinfo' was permanently removed from Sphinx
configuration. Refer to documentation for details.
WARNING: key 'dict' was permanently removed from Sphinx configuration.
Refer to documentation for details.
WARNING: key 'docinfo' was permanently removed from Sphinx
configuration. Refer to documentation for details.
WARNING: key 'dict' was permanently removed from Sphinx configuration.
Refer to documentation for details.
WARNING: key 'docinfo' was permanently removed from Sphinx
configuration. Refer to documentation for details.
WARNING: 13 more warnings skipped.
WARNING: index 'delta1', source 'delta': no kbatch configured in the
index, sql_query_killlist in the source IGNORED!
WARNING: index 'delta1' source 'delta': no kbatch targets configured;
kill-list IGNORED!
WARNING: key 'docinfo' was permanently removed from Sphinx
configuration. Refer to documentation for details.
WARNING: key 'dict' was permanently removed from Sphinx configuration.
Refer to documentation for details.
WARNING: key 'docinfo' was permanently removed from Sphinx
configuration. Refer to documentation for details.
WARNING: key 'dict' was permanently removed from Sphinx configuration.
Refer to documentation for details.
WARNING: key 'docinfo' was permanently removed from Sphinx
configuration. Refer to documentation for details.
WARNING: 13 more warnings skipped.

I'm using piler version 1.3.5 ans sphinxsearch 3.1.1

Thanks for your help.

BKH

Re: SMTP-Port - Change / Checks vie WebGUI

[sj]

Hello,

check out config.php, and find $health_smtp_servers variable at the end 
of it, and fix it.


PS: I'll move this variable somewhere above in config.php, so it can be 
overridden in

config-site.php in the future.

Janos

On 2019-08-22 10:35, BKH wrote:

Hello,

i changed piler SMTP-Port to a self chosen one.
Within healtch check (WebGUI) piler shows ein error: "SMTP-Status:
piler: ERROR"
Where to modify those checks?

Thanks,
BKH

Re: Ubuntu deb download link broken

[sj]

Hello,

I don't think it's broken. The link opens just fine.

Janos

On 2019-06-27 16:11, Yuchen Guo wrote:

This link
https://bitbucket.org/jsuto/piler/downloads/piler_1.3.5~bionic-f2e4cb1_amd64.deb
is broken, please fix.

Regards

YC

Re: Applying Piler for a large / deep folder-hierarchy archive.

[sj]

Hello Arnold,

well, I'd say give piler a shot and see it for yourself if it works
as expected for your use case.

And don't let my personal opinion ("rarely used") misguide you.

Janos

On 2019-06-26 20:39, Arnold Opio Oree wrote:

Hello to you all,

I'd like to ask about my intended application scenario for Piler to
create a principally folder-hierarchy navigated archive for our
enterprise emails, which are treated as live data rather than archives
for compliance or occasional / reactive retrieval.

The data is presently not that large (a few gigabytes), but it is
expected to grow rapidly. Up to this stage the data has been contained
in a Microsoft Exchange mailbox (2013), and then in an offline PST. The
move to the offline PST was by necessity, as the large number of
folders, and depth of hierarchy to my best understanding caused the
exchange server / outlook / evolution mail clients to begin to
malfunction. To cope with this the archive was broken up and the bulk
stored in the offline PST and the most active components stored in
online Exchange mailboxes.

I would like to know that Piler can (once Outlook PST data is imported)
deal with a large number, and deep hierarchy of folders?

The second point I would like to ask about is Piler's folder support (
http://www.mailpiler.org/wiki/current:folder-support). In the
documentation, it says that the feature is rarely used, and is normally
not needed. Given this, is it likely to be suitable for the application
scenario as above? And secondly will the folder hierarchy created by
Piler's folder support be accessible through an IMAP connection, or
will the folders only exist through the GUI?

I will really appreciate any help and advice you can give.

Best regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies
78A Hatfeild Mead
London
SM4 5PF

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.

Re: AW: Downloaded files corrupt

[sj]

OK, I see. I've just checked the latest build with some test emails,
and I couldn't notice the problem you mentioned. Though I checked with
images only as any corruption can be easily spotted.

Btw. can you show me a non sensitive corrupted attachment?
Perhaps via a filesharing site or similar.

Janos

On 2019-04-30 13:06, Katterl Christian wrote:

Hello,

in config-site.php:

$config['DEFAULT_LANG'] = 'en';

Therefore, it should be english already.

BR, Christian

VON: Janos SUTO 
GESENDET: Dienstag, 30. April 2019 11:41
AN: Piler User 
BETREFF: Re: Downloaded files corrupt

Hello,

what language file do you use?

Can reproduce the issue using English?

Janos

-

FROM: Katterl Christian
SENT: Tue Apr 30 05:45:59 GMT+02:00 2019
TO: "piler-user@list.acts.hu"
SUBJECT: Downloaded files corrupt

Hello,

when I try to download archived files (attachments), I noticed, that
the files are corrupted after download.

They are 1 byte larger than the original.

I found this here:

https://bitbucket.org/jsuto/piler/issues/949/corrupted-files-are-downloaded-from-web-ui


I even tried the language-files of the current master-tree, where I
assume, the issue is fixed?

BR, Christian

CHRISTIAN KATTERL
Teamleader Technical IT

ASAMER BAUSTOFFE AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
TEL  +43 50 799 - 2511
MOBILE +43 664 811 54 99
EMAIL c.katt...@asamer.at
WWW.ABAG.AT [1]

This message is confidential. It may not be disclosed to, or used by,
anyone other than the addressee. If you receive this message by
mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334



Links:
--
[1] https://www.abag.at

Re: Upgrade Instructions 1.3.5

[sj]

Hello Christian,

the official upgrade guide mentioned how to upgrade from 1.3.0 to 1.3.4.
Since 1.3.5 is a minor bugfix release, I've updated it as "from 1.3.0 to 
1.3.5".


http://www.mailpiler.org/wiki/current:upgrade

Let me know if you have any questions.

Janos



On 2019-04-29 08:46, Katterl Christian wrote:

Hello,

is there any special upgrading instructions for 1.3.x to 1.3.5?

On the website, I could not find info for this?

BR, Christian

CHRISTIAN KATTERL
Teamleader Technical IT

ASAMER BAUSTOFFE AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
TEL  +43 50 799 - 2511
MOBILE +43 664 811 54 99
EMAIL c.katt...@asamer.at
WWW.ABAG.AT [1]

This message is confidential. It may not be disclosed to, or used by,
anyone other than the addressee. If you receive this message by
mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334


Links:
--
[1] https://www.abag.at

Re: Timestamp Authority "disbanded", what to do now?

[sj]

Hello Frank,

well, I think your only option is to re-stamp existing emails,
since you can't really use them to validate without the (now 
disappeared)

TSA authority who issued them.

Janos

On 2019-04-13 23:33, Frank Schmitz wrote:

seems like the timestamping authority I used "disbanded" and kind of
tells me to go elsewhere for timestamping.

Sure, I can configure another TSA to get NEW timestamps, but what
about all the old timestamps in my archive?
Do I have to restamp every mail in the archive?
What's the best way to do that?
Can I simply add another certificate to be able to verify older mails
with the old timestamps?
Anyone ever had this problem before?

Thankful for any advice...

Re: Message Failed Verification

[sj]

Hello Marcelo,

check if you have copied piler.key, piler.conf from old installation.
(Based on the new messages displayed properly I assume these files and
the piler binaries have proper ownership and permissions.)

Janos

On 2019-02-23 20:02, Marcelo Machado wrote:

Hello everyone.

After migrate piler to a new hardware, my old messages are showing
"Message Failed Verification" and the header/body empty. New messages
are normal.

Someone can help me?

Marcelo Gomes

Re: Rentention-rules and pilerpurge

[sj]

Hello Marina,

On 2018-11-05 13:35, Frau Marina Diezler wrote:


I turned on periodic purge in the backend so that calling pilerpurge
is allowed to delete anything at all. We created some retention rules,
for example all mails sent from our gitlab-instance on another host
should be deleted after 30 days. We added some archiving rules saying
that everything mared as spam or containing \*SPAM\* in the subject
should  be ignored and that the existing spam should be deleted via
retention rule following the same identification rules and giving a
retention time of 1 day. But I can still see "fresh" Spam coming into
the archive and the old spam-messages still being held where they are.


note that the rules (both archiving and retention) affects only new
incoming emails. Already stored emails are not affected.




When I started pilerpurge it deleted only about 100 messages which is
way to few. The retention rules  should meet thousands of messages and
in addition all emails older than 3561 days should be deleted too but
the amount of pilerpurge'd mails so far wasn't so big.

If I try to pilertest an arbitrary message then I never get readable
infos  but always something like this:

root@piler:/var/piler/store/00/5a1/3d/38# pilertest
40005a105109082cef0400aae4733d38.m
locale: de_DE.UTF-8
build: 955
parsing...
post parsing...
message-id: null /
74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b
from: * ()*
to: * ()*
reference: **
subject: **
body: **
sent: 1541417598, delivered-date: 0
hdr len: 0
body digest:
d10a0fcad8f1fe6dee72f70a4d9888604ddfaf3e98b0309ebe23ecabdcd6c542
rules check: (null)
folder: 0
retention period: 1856863998
attachments:
direction: 0
spam: 0
NOT IN mydomains

Is this supposed to be like this?


no. Anything in /var/piler/store is an encrypted and compressed file,
pilertest can't handle it. Instead use pilerget to rertieve it, eg.
pilerget 40005a105109082cef0400aae4733d38 > /tmp/1.eml

and then analyze it with pilertest:

pilertest /tmp/1.eml





Do you have any idea what I should
do  so that more messages are being purged? Especially the retention
period seems suspect I think...


For already archived spam, you should identify them in the metadata 
table,
eg. check the subject field. Then set the retained column for such 
messages
only to the past, eg. a yesterday timestamp, and pilerpurge will take 
care

of them.

Note that such a late job will be slow and pretty ineffective. As you 
stated
above: it's best to specify both archiving and retention rules before 
the 1st

message arrives.



And another question is: Do I understand this correctly: purged mails
still show up forever in the browser backend and only disappear if I
rebuild the index via command line? Is there a way to find out in


No. Once pilerpurge has done, it sets the 'deleted' column to 1 which
would participate for the sphinx k-list (kill list) to suppress the
already purged emails from the gui.

Of course, rebuilding the index is another option since the reindex tool
honors the deleted column.


Janos

Re: High CPU load of mysqld - still after upgrade to 1.3.4

[sj]

Hello Mario,

it's great news :-)

Janos

On 2018-10-10 20:42, Mario Pastoor wrote:


thank you for your're quick reply. I've checked the index last weekend
and watched the behavior of the machine for the last days and I'm lad 
to
say that CPU load is back to a normal level. (see the attached images) 
:-)


Problem was that our previous version was 1.2.0-pre1 and not the final.
To be precise it was this id :
https://bitbucket.org/jsuto/piler/src/20ed9b9e6d87?at=fixed_id

And the index in that version for the attachments table looked like :
create index `attachment_idx` on `attachment`(`piler_id`);
create index `attachment_idx2` on `attachment`(`sig`);
create index `attachment_idx3` on `attachment`(`ptr`);

Here is what I've done to fix it (if anyone else needs this) :
DROP INDEX `attachment_idx` on `attachment`;
DROP INDEX `attachment_idx2` on `attachment`;
DROP INDEX `attachment_idx3` on `attachment`;

create index `attachment_idx` on `attachment`(`piler_id`);
create index `attachment_idx2` on `attachment`(`sig`, `size`, `ptr`);
create index `attachment_idx3` on `attachment`(`ptr`);

OPTIMIZE TABLE attachment;

I'm not sure if the "OPTIMIZE TABLE attachment;" is really needed but 
as

it won't do any harm I ran it in addition.


So, this one is solved.

I've some minor other problems regarding to the webui but that's not
that important right now. As soon as I have some time I'll report here
to the list.

Re: Upgrading from old piler to new

[sj]

Hello Michael,

On 2018-10-07 13:03, Michael Mansour wrote:


I'd like to look at exporting emails from the old and into the new,
based on emails for all the domain instead of defining email
addresses.


well, pilerexport can do that. Go to a directory where piler has write
access, and expect many eml files in the directory. Perhaps it's better
to export a day/week/month or so emails at a time, zip them, and 
transfer

them to new host to import.


As another question, will the import honour the _new_ retention times
on the domain? the old ones I didn't put a retention time on one or
two of the domains, on the new server I wish to set a limit so that
potentially emails can be pruned after 5 years (for some domains).


pilerimport uses the settings on the given host ('new' in your case),
so it's possible to set 5 years retention. Note that pilerexport doesn't
export the set retention time.

Janos

Re: How to properly enable memcached ?

[sj]

On 2018-10-07 13:00, Michael Mansour wrote:


I've tried to search through the documentation for this but can't find
anything definitive.

I have installed a new 1.3.4 Mail Piler server. I have memcached
installed and running, have the PHP PECL Memcached install in PHP
also.

From the docs:

http://www.mailpiler.org/wiki/current:post-install?s[]=%2Amemcache%2A

it seems I just need to add:

$config['MEMCACHED_ENABLED'] = 1;

into the config-site.php file?

Please advise if true and / or if anything else needs to be done.


well, I've been giving some after thoughts to memcached, and still not
decided how to use it a smart way. I'll keep it on my todo list.

Janos

Re: From logout back to login

[sj]

Hello Michael,

On 2018-10-07 13:32, Michael Mansour wrote:


I noticed I made a small mod in your logout.php script when
implementing years ago, I didn't like how it stays on the logout and
you have to manually click the back link.

So I modified the bottom of your logout.php to:

...

}

 header("Refresh: 2;index.php");

 ?>

which performs a refresh to index.php.

You might like to add this small code to your next version?


well, I like to show that the user has actually successfully logged out.
However, it's possible to redirect the user to the login page after a
few seconds.

I'll add the feature to the template file for logout.

Janos

Re: High CPU load of mysqld - still after upgrade to 1.3.4

[sj]

Hello Mario,

On 2018-10-05 21:03, Mario Pastoor wrote:


first of all a huge "thank you" for this piece of software, Janos! :-)


you are welcome :-)

Well, it's a nice troubleshooting anyway. The issue is that the select
query may not use the proper index, so it takes more and more time to
get the result.

For starters, I suggest to stop piler, then run explain against the sql
query:

explain SELECT `id` FROM `attachment` WHERE
`sig`='ade5de3ad755a61885b7c2c41dc8101794743324ce06f411a926ca9bc705d401'
AND `ptr`=0 AND `size`=5044;

I assume it involves a full table scan which is slow.

Check if you have the following indices on table attachment:

create index `attachment_idx` on `attachment`(`piler_id`);
create index `attachment_idx2` on `attachment`(`sig`, `size`, `ptr`);
create index `attachment_idx3` on `attachment`(`ptr`);

Add if either of them is missing, then explain command should return 
something like


MariaDB [piler]> explain select * from attachment where 
sig='167bf31b0167aa4e74659a93735b2f66de46c327af080ece73b770c8f0b8b4d2' 
and ptr=0 and size=1182;

+--+-++--+-+-+-+---+--+---+
| id   | select_type | table  | type | possible_keys 
  | key | key_len | ref   | rows | Extra 
|

+--+-++--+-+-+-+---+--+---+
|1 | SIMPLE  | attachment | ref  | 
attachment_idx2,attachment_idx3 | attachment_idx2 | 270 | 
const,const,const |1 | Using index condition |

+--+-++--+-+-+-+---+--+---+
1 row in set (0.001 sec)


Let me know how it goes.


Janos


I'm chasing a mysqld cpu load problem caused by queries from piler 
right

now.
Hope that someone can help me on this.

Here's the full story :

I've installed our piler-system may 2015 on debian and it worked really
well over the past years!
As time passed by I recognised high CPU load peaks that grew during 
this

year. (see attached graphs - piler_cpu_load_0_1year /
piler_cpu_load_0_1week_may_2018)

A few hardware details :
Bare metal installation
HP ProLiant DL380 G6
2 x Intel Xeon 4-Core-CPU X5560  @2.80GHz
24GB RAM installed
SAS - Direct attach storage - RAID5

As you can see in "piler_cpu_load_0_1week_may_2018" these peaks occure
during normal business days/hours when there's a lot of e-mail traffic.

To avoid "have you already tried the current version?" I upgraded piler
from 1.2.0 build to current 1.3.4 yesterday and hoped this would fix 
the

high load problem. But you guess it ... it did not ;-)

I poked around a bit and the source of the hight load was always the
mysqld process. Same with 1.2.0 and 1.3.4

top output in that situation looks like this  :

top - 11:30:49 up  3:31,  3 users,  load average: 9,81, 9,91, 7,76
Tasks: 254 total,   2 running, 252 sleeping,   0 stopped,   0 zombie
%Cpu(s): 64,2 us,  0,9 sy,  0,0 ni, 34,8 id,  0,0 wa,  0,0 hi,  0,1 si,
 0,0 st
KiB Mem : 24680092 total,  5910708 free,  2615884 used, 16153500 
buff/cache


  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ 
COMMAND
  719 mysql 20   0 2915788 2,224g  16948 S  1037  9,4 709:44.10 
mysqld


MariaDB version info output :
mysql -V
mysql  Ver 15.1 Distrib 10.1.26-MariaDB, for debian-linux-gnu (x86_64)
using readline 5.2
mysqld -V
mysqld  Ver 10.1.26-MariaDB-0+deb9u1 for debian-linux-gnu on x86_64
(Debian 9.1)

Then i had a look what queries run during that load period :
I used "SHOW FULL PROCESSLIST" to view the running queries.

The following query showed up 10 times (see attached
MariaDB_process_list_during_load.txt):

SELECT `id` FROM `attachment` WHERE
`sig`='ade5de3ad755a61885b7c2c41dc8101794743324ce06f411a926ca9bc705d401'
AND `ptr`=0 AND `size`=5044


10 rows in set (18.17 sec) is the result running that query when 
there's

no load.
These queries stop apearing a few minutes after stopping piler
(rc.piler) and mysqld comes back to normal.

The I tried to figure out what
`sig`='ade5de3ad755a61885b7c2c41dc8101794743324ce06f411a926ca9bc705d401'
might be and it seems to be our logo that is attached / embeeded in
nearly every mail.

Question is .. why is piler running this query?
And can performance for that queries be optimized?
An aditional index needed or may our index is broken/wrong?
(see attached MariaDB_attachment_index.txt)

Any help would be appreciated :-)

Have a nice weekend folks

Mario

Re: Upgrading from old piler to new

[sj]

Hello Michael,

I recommend to upgrade to the latest stable version 1.3.4.

One possible way to do it is deploying centos 7 x64 with php 7
on a new host, then exporting all emails from old archive, and
import them on new archive.

When you have double checked that new archive works properly, you
may decommission the old host. Or perhaps wait a week/month before
doing so, just in case.

Janos

On 2018-10-06 12:27, Michael Mansour wrote:

Hi,

I've downloaded 1.2.0 and couldn't find much in there as to
installing, it's been years since I last done this so I went to:

http://www.mailpiler.org/wiki/current:installation

I run on CentOS 6.10, which comes with php 5.3.3 by default. The
install docs show php 7.x is needed, which isn't a major problem but
now stops me dead in my tracks.

Do I upgrade the PHP on CentOS 6 to PHP v7.x or do I install a CentOS
7 machine and put that onto PHP 7.x ?

I'd rather install a CentOS 7 machine for longevity, but this poses
the next problem:

1. do I install 1.1.1 on CentOS 7, migrate the 1.1.1 files, DB, etc to
the CentOS 7 server, then upgrade 1.1.1 to 1.2.0, then to further
versions, or

2. do I install the CentOS 7 machine with the latest Piler 1.3.4, and
then work out how to migrate from the CentOS 6 Piler 1.1.1 machine to
the CentOS 7 1.3.4 Piler machine?

Please advise your thoughts and recommendations.

Thanks.

Michael.

On 6/10/2018 7:02 PM, Michael Mansour wrote:


Hi Janos,

Thank you, I found the web page for it here:
http://www.mailpiler.org/wiki/current:upgrade

How do I find out which version I am running?

I think I'm running 1.1.1, because that's the only archive I've
found on the server when I installed it 28 April 2015.

But I want to make sure before performing the upgrade.

Thanks.

Michael.

On 5/10/2018 9:19 PM, Janos SUTO wrote:


Hello,

well, the official site has a section for the upgrade. It should
work.

It's worth to backup the MySQL database, your configs and the
piler.key.

Janos

-
FROM: Michael Mansour
SENT: Fri Oct 05 01:47:18 GMT+02:00 2018
TO: Piler User

SUBJECT: Upgrading from old piler to new

Hi,

I have an older Mail Piler version and would like to update to the
latest.

Are there any upgrade docs on-line or similar that I may look at
so the
upgrade is smooth?

Thanks.

Michael.


 [1]
Virus-free. www.avast.com [1]



Links:
--
[1]
https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=emailclient

Re: Archiving rules not applied

[sj]

Hello Carsten,

have you restarted the piler daemon? It reads the rules when starting.

Janos

On 2018-06-01 11:17, Carsten Pohle wrote:

Hi,

I've defined a couple of archiving rules (so that matching e-mail would
NOT be archived). When testing them with pilertest, they match as
intended. However, the mails are still archived. Did I miss a
configuration option to enable archiving rules at all?

Regards

Carsten

Re: yet another gdpr related email

[sj]

Hello Patrick,

On 2018-05-27 21:14, Wagner, Patrick wrote:


at the end of the gdpr-related-notes document you mention three
specific features: delete feature / purging feature / legal hold
feature. I'm aware of the documentation for pilerpurge on your
website, but what about the other two features?


legal hold: you may specify email addresses that would be excluded
from purging even if (some of) those emails are aged.

deletion: an auditor is able to purge a message from the archive
(by fixing its retention value in the background) if ENABLE_DELETE
is set to 1 in config-site.php

Janos

yet another gdpr related email

[sj]

Dear piler users,

now that GDPR is on us, we all got tons of updated privacy policy emails
from several places. I've also created the privacy policy for the piler
sites, please check http://www.mailpiler.org/wiki/privacy-policy

This link refers to https://mailpiler.com/privacy-policy/ which applies
to both mailpiler.com and (www.)mailpiler.org. The .org site has some
additional notes regarding the mailing list and bitbucket.

You may check http://www.mailpiler.org/wiki/current:gdpr-related-notes
as well. This latter link provides a few notes you should know if you
have an email archive. I assume it takes all of us.

It describes what data the piler archive stores, what security measures
it supports, etc.

Let me know if you have any questions regarding either of these topics.

Janos

Re: Running pilter.pl as a service

[sj]

Hello,

if /var/piler/store/00 does not exist, then create it, and chown it to 
piler.


Janos


On 2018-04-25 16:09, John Huong wrote:

Hi,

I'm seeing these in the syslog.

Apr 26 00:07:33 archive pilerimport[12512]:
40005ae0a7cf1d86672c005304895f97: mkdir
/var/piler/store/00/5ae/5f/97: error=No such file or directory
Apr 26 00:07:33 archive pilerimport[12512]:
40005ae0a7cf1d86672c005304895f97: cannot open:
/var/piler/store/00/5ae/5f/97/40005ae0a7cf1d86672c005304895f97.a1
Apr 26 00:07:33 archive pilerimport[12512]:
40005ae0a7cf1d86672c005304895f97: error storing attachment:
40005ae0a7cf1d86672c005304895f97.a1
Apr 26 00:07:33 archive pilerimport[12512]:
40005ae0a7cf1e217a6400fbba2a1f04: mkdir
/var/piler/store/00/5ae/1f/04: error=No such file or directory
Apr 26 00:07:33 archive pilerimport[12512]:
40005ae0a7cf1e217a6400fbba2a1f04: cannot open:
/var/piler/store/00/5ae/1f/04/40005ae0a7cf1e217a6400fbba2a1f04.m
Apr 26 00:07:33 archive pilerimport[12512]:
40005ae0a7cf1e217a6400fbba2a1f04: error storing message:
40005ae0a7cf1e217a6400fbba2a1f04.m
Apr 26 00:07:33 archive pilerimport[12512]:
40005ae0a7cf1e86cfcc00aec29de87b: mkdir
/var/piler/store/00/5ae/e8/7b: error=No such file or directory
Apr 26 00:07:33 archive pilerimport[12512]:
40005ae0a7cf1e86cfcc00aec29de87b: cannot open:
/var/piler/store/00/5ae/e8/7b/40005ae0a7cf1e86cfcc00aec29de87b.a1
Apr 26 00:07:33 archive pilerimport[12512]:
40005ae0a7cf1e86cfcc00aec29de87b: error storing attachment:
40005ae0a7cf1e86cfcc00aec29de87b.a1

Hope you can advise on the next step. The folder permissions seems to
be correct.


On Wed, Apr 25, 2018 at 8:23 PM, Janos SUTO  wrote:

Hello,

check the logs. If piler encounters a problem, it will syslog it.

Janos

From: John Huong
Sent: Wed Apr 25 09:11:27 GMT+02:00 2018
To: Piler User
Subject: Re: Running pilter.pl as a service



Hi I got pilter.pl running in the background.

I'm not sure if I'm doing things correctly but when i run pilterimport
it generates errors instead.

pilerimport -d /var/piler/imap/cur -D
failed to import: /var/piler/imap/cur/1524639972-820035-4378-EMYFRKOW
(id: 40005ae029c0076ab844005dcd3da30b)
error importing: '/var/piler/imap/cur/1524639972-820035-4378-EMYFRKOW'
failed to import: /var/piler/imap/cur/1524639776-195775-4378-PBG39UA4
(id: 40005ae029c00866fa64006ff10703e0)
error importing: '/var/piler/imap/cur/1524639776-195775-4378-PBG39UA4'
failed to import: /var/piler/imap/cur/1524640059-82319-4380-LWGG0P2Z
(id: 40005ae029c008988214001a841dbe7e)
error importing: '/var/piler/imap/cur/1524640059-82319-4380-LWGG0P2Z'
failed to import: /var/piler/imap/cur/1524639684-292356-4376-QMNNYITL
(id: 40005ae029c008c268a400752fdace29)
error importing: '/var/piler/imap/cur/1524639684-292356-4376-QMNNYITL'
processed:   0

On Sat, Apr 21, 2018 at 9:25 PM,   wrote:




 Hello,

 well, pilter.pl can run in the foreground, so any service
 manager should be able to handle it. You should check some
 examples of your favourite implementation.

 Try this:
 
https://wiki.ubuntu.com/SystemdForUpstartUsers#Example_Systemd_service


 Janos

 On 2018-04-19 22:16, John Huong wrote:



 HI,

 Can anyone share their experience or give pointers how to run
 pilter.pl [1] as a service in Ubuntu 16.04? Think they are using
 systemd.

Re: Running pilter.pl as a service

[sj]

Hello,

well, pilter.pl can run in the foreground, so any service
manager should be able to handle it. You should check some
examples of your favourite implementation.

Try this: 
https://wiki.ubuntu.com/SystemdForUpstartUsers#Example_Systemd_service


Janos

On 2018-04-19 22:16, John Huong wrote:

HI,

Can anyone share their experience or give pointers how to run
pilter.pl [1] as a service in Ubuntu 16.04? Think they are using
systemd.

new feature: support for an external dashboard

[sj]

Dear piler users,

I've just committed a patchset to the master branch to support
an external dashboard. You may check it how it looks on the demo
site: http://demo.mailpiler.org, use admin@local as username and
pilerrocks as the password.

The commit is actually very simple: if an url is set, then it's
displayed in an iframe in monitor / health menu.

The demo site is fixed to display a snapshot of a grafana dashboard
that's why it displays only a limited time range. (Normally it
would be a grafana link instead of a snapshot).

Of course any monitoring system could be used, provided that it can
be shown in an iframe.

You may use your own monitoring dashboard, or you may sign up for
any 'dashboard as a service' provider. As you may have figured it out
already, I'm about to launch such a service, too.

The details and the pricing are still being polished. Probably there
will be a basic plan consisting a predefined dashboard with a 1 month
retention of the values, and a pro plan with a longer retention 
supporting

slack/email alerts as well the option to customize the dashboard, eg.
add new graphs, annotations, etc.

Janos

Re: Version 1.3.4

[sj]

Hello Christian,

On 2018-03-20 11:23, Katterl Christian wrote:


i noticed, that there's an updated version of mailpiler - version
1.3.4.

Unfortunately i could not find recent release-notes or upgrading
instructions?

The release notes included in the archive are stopping with 1.3.0 and
the upgrade instructions
(http://www.mailpiler.org/wiki/current:upgrade) are stopping with
upgrade 1.3.0 to 1.3.2.

Where can i find the proper information?


well, I was short on time, that's why I didn't make a proper 
announcement,

and not updated the upgrade docs yet.

Since it's a bugfix release, simply updating the binaries and the gui 
files

will do the trick, no need to fix the database schema.

With that being said the database schema did change: I've extended a few
columns, and changed the type for one or two column, because I managed 
to

see some installations requiring longer from or to addresses, and having
flawed encoding in From/To columns. I'll explain these in the upgrade 
docs

soon.

Most sites, however, should be fine with current database layout.

Janos

A survey on commercial services and products for piler

[sj]

Dear piler users,

I'd like you to visit http://www.mailpiler.org, and take a short
(really!) survey helping me to work on some piler related commercial
services, and a commercial version of piler. A small popup window
opens asking whether you want to give feedback.

If your first automatic move is just like mine (=hit the close icon
on it), but you change your mind later, then you may find the survey
link on the top of the wiki.

The last question is a free form text area for comments. If you want
a response, then either don't forget to add your email, or you may
send your comments to my email directly.

Thanks for helping, your opinion matters!

Janos

Re: AW: Problem with distribution groups

[sj]

Hello,

On 2018-02-26 16:49, Katterl Christian wrote:


I reindexed everything and now it’s working for some reason.


ok, fine. I've tested the parser with your email, and it was almost
fine: it found all the email addresses, however to recipient domain
list was empty. I've fixed this, and pushed the commit to the master
branch.

Janos

Re: Internal messages

[sj]

On 2018-02-15 20:09, Marcelo Machado wrote:

I need to archive only messages from my domain to my domain.

This is possible?


it's difficult, because the tre regex doesn't support negating logic.
The best would be if you could forward only emails meant to be archived
to piler.

Janos

Re: AW: BCC not shown

[sj]

ok, I see. Try retrieving this email from archive, eg.
pilerget 40005a82074b237e7e9400cba0225cf3 > /tmp/1.eml, then
check the email, especially the journaling stuff. Your email
address (even if it's bcc) should be somewhere in the headers.

If so, then run pilertest 1.eml, and check if your (bcc) address
is among the recipients.

Janos

On 2018-02-12 23:19, Christian Röser - PELMA wrote:

Glad to hear that.  I did some extensive tests and I'm thinking that
we are dealing here with some fucked up (sorry) exchange related
problem. The address of my "colleague" is a shared folder where I also
have full access rights. But that doesn't seems to be the problem
because I also send a mail to another shared folder with my address in
bcc - worked. Then I gave me access rights and send another mail -
still worked. It seems to be something exclusive with this address but
I have no idea what's wrong.


Regarding the information you wanted:

MariaDB [piler]> select * from rcpt where id=80;
++-++
| id | to  | todomain   |
++-++
| 80 | c.roe...@saftig.net | saftig.net |
| 80 | i...@saftig.net   | saftig.net |
++-++
2 rows in set (0.00 sec)


Connected with sphinx and query with my address:

MySQL [(none)]> SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('
(@from cXroeserXsaftigXnet | @to cXroeserXsaftigXnet) ') ORDER BY
`sent` DESC LIMIT 0,20 OPTION max_matches=1000;
+--+
| id   |
+--+
|   79 |
|   77 |
|   76 |
|   75 |
|   74 |
|   72 |
|   71 |
|   70 |
|   69 |
|   66 |
|   65 |
|   57 |
|   56 |
|   54 |
|   55 |
|   52 |
|   48 |
|   43 |
|   41 |
|   39 |
+--+
20 rows in set (0.00 sec)


Query with the address of the shared folder:

MySQL [(none)]> SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('
(@from itXsaftigXnet | @to itXsaftigXnet) ') ORDER BY `sent` DESC
LIMIT 0,20 OPTION max_matches=1000;
+--+
| id   |
+--+
|   80 |
|   73 |
|   74 |
|   70 |
|   44 |
|   36 |
|   35 |
|   28 |
|   29 |
+--+
9 rows in set (0.00 sec)


And here is the entries from the log regarding this specific message:

Feb 12 22:29:36 mailpiler piler-smtp[549]: connected from
10.194.0.2:16698 on descriptor 6 (active connections: 1)
Feb 12 22:29:36 mailpiler piler-smtp[549]: [54B blob data]
Feb 12 22:29:36 mailpiler piler-smtp[549]: [77B blob data]
Feb 12 22:29:36 mailpiler piler-smtp[549]: [46B blob data]
Feb 12 22:29:36 mailpiler piler-smtp[549]: sent: 250 Ok
Feb 12 22:29:36 mailpiler piler-smtp[549]: [67B blob data]
Feb 12 22:29:36 mailpiler piler-smtp[549]: sent: 250 Ok
Feb 12 22:29:36 mailpiler piler-smtp[549]: [27B blob data]
Feb 12 22:29:36 mailpiler piler-smtp[549]: sent: 354 Send mail data;
end it with .
Feb 12 22:29:36 mailpiler piler-smtp[549]: received: 5VQK1KDSHAENMIU1,
from=, size=8361, client=10.194.0.2
Feb 12 22:29:36 mailpiler piler-smtp[549]: sent: 250 OK 
<5VQK1KDSHAENMIU1>

Feb 12 22:29:36 mailpiler piler-smtp[549]: [27B blob data]
Feb 12 22:29:36 mailpiler piler-smtp[549]: sent: 221
mailpiler.saffig.saftig.net Goodbye
Feb 12 22:29:36 mailpiler piler-smtp[549]: disconnected from
10.194.0.2 on descriptor 6 (0 active connections)
Feb 12 22:29:37 mailpiler piler[554]:
40005a82074b237e7e9400cba0225cf3: journal rcpt: 'i...@saftig.net '
Feb 12 22:29:37 mailpiler piler[554]:
40005a82074b237e7e9400cba0225cf3: hdr_len: 594, offset: 3
Feb 12 22:29:37 mailpiler piler[554]:
40005a82074b237e7e9400cba0225cf3: touch
1d144d5a090b040ea90a909a1b06cf8b7b81d96e1b88e0d01c8949ead45a887b OK
(<20180212212922.14342101...@smtp02.mail.de>)
Feb 12 22:29:37 mailpiler piler[554]:
40005a82074b237e7e9400cba0225cf3: stored
'40005a82074b237e7e9400cba0225cf3.m' 8361/3896 bytes
Feb 12 22:29:37 mailpiler piler[554]:
40005a82074b237e7e9400cba0225cf3: stored 1 recipients, rc=0
Feb 12 22:29:37 mailpiler piler[554]:
40005a82074b237e7e9400cba0225cf3: stored metadata, rc=0
Feb 12 22:29:37 mailpiler piler[554]: 1/5VQK1KDSHAENMIU1:
40005a82074b237e7e9400cba0225cf3, size=8361/3896, attachments=0,
reference=, message-id=<20180212212922.14342101...@smtp02.mail.de>,
retention=4000, folder=0, delay=0.0219, status=stored
Feb 12 22:29:45 mailpiler piler-smtp[549]: connected from
10.194.0.2:16706 on descriptor 6 (active connections: 1)
Feb 12 22:29:45 mailpiler piler-smtp[549]: [54B blob data]
Feb 12 22:29:45 mailpiler piler-smtp[549]: [77B blob data]
Feb 12 22:29:45 mailpiler piler-smtp[549]: [46B blob data]
Feb 12 22:29:45 mailpiler piler-smtp[549]: sent: 250 Ok
Feb 12 22:29:45 mailpiler piler-smtp[549]: [67B blob data]
Feb 12 22:29:45 mailpiler piler-smtp[549]: sent: 250 Ok
Feb 12 22:29:45 mailpiler piler-smtp[549]: [27B blob data]
Feb 12 22:29:45 mailpiler piler-smtp[549]: sent: 354 Send mail data;
end it with .
Feb 12 22:29:45 mailpiler piler-smtp[549]: received: R10MNQ5X6TSNS4DS,
from=, size=8279, client=10.194.0.2
Feb 12 22:29:45 mailpiler piler-smtp[549]: sent: 250 

Re: Problem with S/MIME signed mail, Piler 1.31 on Ubuntu LTS 16.04

[sj]

Hello Daniel,

On 2018-01-31 17:51, Daniel Abegglen wrote:


I've problems with S/MIME signed mails. There is no body visible. If I
download the .eml file there is not recipient or sender, nor a
subject, just header and the S/MIME-part.


can you share such a signed and non sensitive email (the complete eml 
file)?

Then I'd check what's wrong.

Janos

Re: Sphinx sql_query_killlist not applied ?

[sj]

On 2018-01-25 19:16, Konstantin wrote:


Your query return empty set

I supposed that id=11 should be removed from main1...


no, not at all. You have removed the message with id=11, and
you wondered why you could still see it in the sphinx results?

The answer is that the killlist feature requires the delta index
in the query, just as in the query I gave you.



So do you recommend to update sphinxsearch to the latest version
2.1.X?  but not 2.2, i remember your attention about 32-bit document
IDs...


Don't upgrade it to 2.1.x, it's obsoleted as well. Sphinx has just
released 3.0.1. So I suggest at least the latest 2.2.x series. However,
the upgrade may force you to drop the index data, and reindex 
everything.

So I suggest you to leave 2.0.x for now - if you are happy with it.

Recent piler build, however, require 2.2 or better versions of sphinx

Jano

Re: Sphinx rotation and deleted messages

[sj]

Hello,

On 2018-01-24 13:30, Konstantin wrote:


I have index file with size over 200Gb and want to rotate main1 to
main2 according to the wiki
http://www.mailpiler.org/wiki/current:administering-piler

Will sphinx clean data from main1 (old index) based on killist if i
make change in indexer-main.sh ?


no. If you fixed the main index from main1 to main2 it wouldn't touch
the main1 data. The sphinx killlist works in a different way.

Janos