On Tuesday 02 September 2014 23:12, Steven Morgan wrote:
> Hi J. David,
>
> Thanks for the additional analysis and information. I've been looking at
> this for a bit today. I have opened a ticket in the ClamAV bugzilla system
> to track the issue. The ticket number is 11089. Hope to have an answer
> soon.
>
Have been doing a bit of hacking around with the code.
Have not been able to get the logging functions built in to libclamd
functions such as "cli_dbgmsg" to log reliably.
freshcam and clamd appear to use a call back to intercept and route
their output via their own function "logg". However clamav-milter
seeems to be missing the necessary bits to do this.
I ended up putting direct calls in to fprintf and fflush.
problem is definitely in cl_hash_data. As a work round i have patched
the function cli_md5buff in libclamav/others_common.c to use the
random data directly to create the file name when the call to
cl_hash_data fails.
[root@mailhost-c6 BUILD]# diff -Naur
clamav-0.98.4.orig/libclamav/others_common.c
clamav-0.98.4.rt/libclamav/others_common.c
--- clamav-0.98.4.orig/libclamav/others_common.c 2014-05-21
16:25:05.000000000 +0100
+++ clamav-0.98.4.rt/libclamav/others_common.c 2014-09-08 19:27:00.458404432
+0100
@@ -825,21 +825,41 @@
{
unsigned char digest[16];
char *md5str, *pt;
+ const unsigned char *pt2;
int i;
- cl_hash_data("md5", buffer, len, digest, NULL);
+ if(!(md5str = (char *) cli_calloc(32 + 1, sizeof(char)))) {
+ cli_dbgmsg("cli_md5buff: out of memory\n");
+ return NULL;
+ }
- if(dig)
- memcpy(dig, digest, 16);
+ if ( cl_hash_data("md5", buffer, len, digest, NULL) == NULL ){
+ cli_dbgmsg("cli_md5buff: Call to cl_hash_data failed to return
hash\n");
+ fprintf(stderr, "cli_md5buff:fprint Call to cl_hash_data failed to
return hash\n");
+
+/* In the absence of hash use random data from buffer directly */
+/* this is a nasty cludge based onthe fact that we are on called from
cli_gentmp */
+/* with 16 byes saved from last digest, and 32 bytes of fresh random data */
+ pt = md5str;
+ pt2 = buffer + 16 ;
+ for(i = 0; i < 16; i++) {
+ sprintf(pt, "%02x", *pt2);
+ pt += 2;
+ pt2 += 2;
+ }
- if(!(md5str = (char *) cli_calloc(32 + 1, sizeof(char))))
- return NULL;
+ } else { /* successful hash */
+ fprintf(stderr, "cli_md5buff:fprint using hash returned from
cl_hash_data\n");
- pt = md5str;
- for(i = 0; i < 16; i++) {
+ if(dig)
+ memcpy(dig, digest, 16);
+
+ pt = md5str;
+ for(i = 0; i < 16; i++) {
sprintf(pt, "%02x", digest[i]);
pt += 2;
- }
+ }
+ };
return md5str;
}
@@ -889,11 +909,14 @@
if(!tmp) {
free(name);
- cli_dbgmsg("cli_gentemp('%s'): out of memory\n", mdir);
+ cli_dbgmsg("cli_gentemp('%s'): error from cli_md5buff\n", mdir);
return NULL;
}
snprintf(name, len, "%s"PATHSEP"clamav-%s.tmp", mdir, tmp);
+/* log all names */
+ cli_dbgmsg("cli_gentemp: filename %s"PATHSEP"clamav-%s.tmp\n", mdir,
tmp);
+ fprintf(stderr, "cli_gentemp: filename %s"PATHSEP"clamav-%s.tmp\n",
mdir, tmp);
free(tmp);
return(name);
@@ -915,7 +938,9 @@
free(*name);
*name = NULL;
return CL_ECREAT;
- }
+ } else {
+ cli_errmsg("cli_gentempfd: Createed temporary file %s: %i\n", *name,
*fd);
+ };
return CL_SUCCESS;
}
[root@mailhost-c6 BUILD]#
Diagnostic output from my current version below.
Note it is the call to EVP_get_digestbyname from cli_hash_data that fails.
Reading the manual page on EVP_get_digestbyname, it will only work if
OpenSSL_add_all_digests() or OpenSSL_add_all_algorithms() has been called
first.
It looks like these are only called from cl_initialize_crypto().
Snag as far as I can see cl_initialize_crypto is not called from clamav-milter
LibClamAV Info: cli_hash_data: Called
LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
cli_md5buff:fprint Call to cl_hash_data failed to return hash
cli_gentemp: filename /tmp/clamav-cc4ad42fc23e93aaf82a1acc428307d9.tmp
LibClamAV Error: cli_gentempfd: Createed temporary
file /tmp/clamav-cc4ad42fc23e93aaf82a1acc428307d9.tmp: 11
LibClamAV Info: cli_hash_data: Called
LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
cli_md5buff:fprint Call to cl_hash_data failed to return hash
cli_gentemp: filename /tmp/clamav-6296cff0b0f8e0d462e2a82ede5885ac.tmp
LibClamAV Error: cli_gentempfd: Createed temporary
file /tmp/clamav-6296cff0b0f8e0d462e2a82ede5885ac.tmp: 13
LibClamAV Info: cli_hash_data: Called
LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
cli_md5buff:fprint Call to cl_hash_data failed to return hash
cli_gentemp: filename /tmp/clamav-40f9e43c4dc6f6213d604a6c642bb738.tmp
LibClamAV Error: cli_gentempfd: Createed temporary
file /tmp/clamav-40f9e43c4dc6f6213d604a6c642bb738.tmp: 11
LibClamAV Info: cli_hash_data: Called
LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
cli_md5buff:fprint Call to cl_hash_data failed to return hash
cli_gentemp: filename /tmp/clamav-45633b0b40207939c2acdc5d752f0862.tmp
LibClamAV Error: cli_gentempfd: Createed temporary
file /tmp/clamav-45633b0b40207939c2acdc5d752f0862.tmp: 12
LibClamAV Info: cli_hash_data: Called
LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
cli_md5buff:fprint Call to cl_hash_data failed to return hash
cli_gentemp: filename /tmp/clamav-876a73b889eb4308c4773efa6818933d.tmp
LibClamAV Error: cli_gentempfd: Createed temporary
file /tmp/clamav-876a73b889eb4308c4773efa6818933d.tmp: 12
LibClamAV Info: cli_hash_data: Called
LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
cli_md5buff:fprint Call to cl_hash_data failed to return hash
cli_gentemp: filename /tmp/clamav-6ed0e7970718b603f4125c9c920aecee.tmp
LibClamAV Error: cli_gentempfd: Createed temporary
file /tmp/clamav-6ed0e7970718b603f4125c9c920aecee.tmp: 11
LibClamAV Info: cli_hash_data: Called
LibClamAV Error: cli_hash_data: Call to EVP_get_digestbyname failed
cli_md5buff:fprint Call to cl_hash_data failed to return hash
cli_gentemp: filename /tmp/clamav-0e1024e48ddf5d9245ec42cc366bba55.tmp
LibClamAV Error: cli_gentempfd: Createed temporary
file /tmp/clamav-0e1024e48ddf5d9245ec42cc366bba55.tmp: 12
> Steve
>
> On Sun, Aug 31, 2014 at 5:52 AM, J. David Rye <d....@roadtech.co.uk> wrote:
> > On Thu, 2014-08-21 at 19:22 -0400, Steven Morgan wrote:
> > > Hi Urban,
> > >
> > > I took a look at this code. The real problem is the inability to
> > > create a
> > > temporary file. The second message just results from the return code
> > > of the
> > > function that attempts to create the temp file. We need to find out
> > > why the
> > > temp file creation fails. There should also be a clamav error message
> > > written from: cli_errmsg("cli_gentempfd: Can't create temporary file
> > > %s:
> > > %s\n", *name, strerror(errno)); Can you find this message?
> > >
> > > Otherwise, it is a memory allocation failure for space for the temp
> > > file
> > > name, which seems unlikely.
> > >
> > > Steve
> >
> > I am also seeing this issue. Mostly intermitant but see further down.
> >
> > cli_errmsg wont work if clamav-milter has daemonezed.
> > it only writes to STDERR and the function daemonize closes standard
> > error even if you recompile with CL_DEBUG set.
> >
> > Only way to get is to get the error messages from cli_gentempfd seams
> > to be to uncomment the line
> >
> > "#Foreground yes"
> >
> > In clamav-milter.conf, then run in foreground from command line.
> >
> > As an aside I wonder why cli_gentempfd does not use the function logg()
> > and output to file or syslog depending on configuration file.
> >
> > I am running clamav-milter on a VM. OS is CentoOS 6.5
> > VM has 4 vcpu, and 2GB RAM
> > clamav-milter is version 0.98.4-1.el6.rf installed from rpmforge
> > repository.
> >
> > Looking at he logs if time stamps in syslog for calls to clamav-milter
> > are two seconds or more apart the problem never shows.
> >
> > However if 4 or more messages arrive in two seconds problem always shows
> > up, the failure to create temp file is usually time stamped 2 seconds
> > after the first message in the burst that triggered it.
> >
> > On a sustained burst of traffic pretty much all the messages trip the
> > issue.
> > In a 1 hour period last week when I had a lot of messages due to a
> > different issue. I had 20,000 temp file failures, and 23 messages
> > delivered.
> >
> > [root@mailhost-c6 etc]# clamav-milter
> > --config-file=/etc/clamav-milter.conf.foreground
> > Local socket unix:/var/run/clamav/clamd.sock added to the pool (slot 1)
> > Probe for slot 1 returned: success
> > LibClamAV Error: cli_gentempfd: Can't create temporary
> > file /tmp/clamav-0000000000000000626683ff3a000000.tmp: File exists
> > ERROR: Failed to create temporary file
> > ERROR: Failed to initiate streaming/fdpassing
> > LibClamAV Error: cli_gentempfd: Can't create temporary
> > file /tmp/clamav-0000000000000000626683ff3a000000.tmp: File exists
> > ERROR: Failed to create temporary file
> > ERROR: Failed to initiate streaming/fdpassing
> > LibClamAV Error: cli_gentempfd: Can't create temporary
> > file /tmp/clamav-0000000000000000626683ff3a000000.tmp: File exists
> > ERROR: Failed to create temporary file
> > ERROR: Failed to initiate streaming/fdpassing
> > LibClamAV Error: cli_gentempfd: Can't create temporary
> > file /tmp/clamav-0000000000000000626683ff3a000000.tmp: File exists
> > ERROR: Failed to create temporary file
> > ERROR: Failed to initiate streaming/fdpassing
> > LibClamAV Error: cli_gentempfd: Can't create temporary
> > file /tmp/clamav-0000000000000000626683ff3a000000.tmp: File exists
> > ERROR: Failed to create temporary file
> > ERROR: Failed to initiate streaming/fdpassing
> > Message from <n0r3ply812...@scotland117.wanadoo.co.uk> to
> > <stevensonbros> infected by Heuristics.Phishing.Email.SpoofedDomain
> > Message from <n0r3ply620...@aughamullan.dungannon.ni.sch.uk> to
> > <brett01> infected by Heuristics.Phishing.Email.SpoofedDomain
> > Probe for slot 1 returned: success
> >
> >
> > I think
> >
> > clamav-milter does a lot of initialization, including setting up a
> > structure with a list of function entry points it then calls smfi_main.
> >
> > smfi_main in turn forks one thread for each message, and calls the entry
> > points in the context of the thread.
> >
> > call back to function clamfi_header
> > that calls sendchunk
> > which calls nc_connect_rand
> > which calls cli_gentempfd which prints the EEXISTS errors to stderr
> > shown above.
> >
> > File name looks like it is supposed to be based on a 16 byte MD5 digest
> > printed in hex.
> >
> > cli_gentempfd builds the name by calling cli_gentemp
> > which adds 32 bytes of random data from cli_rndnum to a 16 byte
> > seed
> > passes a pointer the the 48 buffer to cli_md5buff
> > which in turn calls cl_hash_data
> > digest returned is used to create the file name,
> > and update the seed for the next name.
> >
> >
> > Note cli_gentemp is common to all programs in the clam set.
> >
> > Note cl_hash_data is calling openssl library functions to calculate the
> > digest.
> >
> > Note cli_md5buff does not check for an error in cl_hash_data
> >
> > _______________________________________________
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> > http://www.clamav.net/support/ml
>
> _______________________________________________
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> http://www.clamav.net/support/ml
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
