Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[John Abreau]

Do they work on the old server but not on the new server? Or do they fail
on the old server as well?

If they work on the old server, then you can run them on some sample input
and compare the input with the output to see what is being changed. If it's
just mapping html entities, it should be easy enough to recode it in perl
or python.


On Sun, Jul 27, 2014 at 11:08 PM, Bill Horne  wrote:

> I'm moving the Telecom Digest to a new server, and I'm seeking information
> about a binary named "entities", and one named "fixup".  I've found them in
> a script that processes emails into html pages for publications, but the
> script's author isn't available, and neither is working.
>
> AFAICT, "entities" is supposed to replace ASCII characters with HTML
> Entities where needed, and I don't know what "fixup" is supposed to do.
>
> All suggestions welcome.
>
> Bill
>
> --
> E. William Horne
> William Warren Consulting
> 339-364-8487
>
> ___
> Discuss mailing list
> Discuss@blu.org
> http://lists.blu.org/mailman/listinfo/discuss
>



-- 
John Abreau / Executive Director, Boston Linux & Unix
Email j...@blu.org / WWW http://www.abreau.net / 2013 PGP-Key-ID 0x920063C6
2013 / ID 0x920063C6 / FP A5AD 6BE1 FEFE 8E4F 5C23  C2D0 E885 E17C 9200 63C6
2011 / ID 0x32A492D8 / FP 7834 AEC2 EFA3 565C A4B6  9BA4 0ACB AD85 32A4 92D8
___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Bill Ricker]

On Sun, Jul 27, 2014 at 11:18 PM, John Abreau  wrote:

> If it's
> just mapping html entities, it should be easy enough to recode it in perl
> or python.
>

​That script surely already exists​, ready for reuse in some CPAN or other
archive?

(But to be really helpful, it should detect MACscii and convert that too.)



-- 
Bill Ricker
bill.n1...@gmail.com
https://www.linkedin.com/in/n1vux
___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Bill Horne]

On 7/27/2014 11:18 PM, John Abreau wrote:
Do they work on the old server but not on the new server? Or do they 
fail on the old server as well?


No, they work on the old server, but fail on the new. I assume it's a 
permissions issue, but I can't figure out what might cause it.


Bill

--
E. William Horne
William Warren Consulting
339-364-8487

___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Tom Metro]

Bill Horne wrote:
> No, they work on the old server, but fail on the new. I assume it's a
> permissions issue, but I can't figure out what might cause it.

% strace entities

 -Tom

-- 
Tom Metro
The Perl Shop, Newton, MA, USA
"Predictable On-demand Perl Consulting."
http://www.theperlshop.com/
___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Bill Horne]

On 7/28/2014 5:33 PM, Tom Metro wrote:

Bill Horne wrote:

No, they work on the old server, but fail on the new. I assume it's a
permissions issue, but I can't figure out what might cause it.

% strace entities


Tom,

Thanks for the suggestion. Here's a log snippet that may make this issue 
more clear:


moder8@telecom:~/bin$ ls -lh /home/moder8/bin/entities
-rwxrwxr-x 1 moder8 telecom 8.8K Jan 27  2012 /home/moder8/bin/entities

moder8@telecom:~/bin$ ls -lh 
/var/www/html/archives/back.issues/recent.single.issues/I125
-rw-rw-r-- 1 root root 3.1K Jul 28 11:12 
/var/www/html/archives/back.issues/recent.single.issues/I125


moder8@telecom:~/bin$ strace ./entities 
/var/www/html/archives/back.issues/recent.single.issues/I125
execve("./entities", ["./entities", 
"/var/www/html/archives/back.issu"...], [/* 22 vars */]) = -1 ENOENT (No 
such file or directory)
write(2, "strace: exec: No such file or di"..., 40strace: exec: No such 
file or directory

) = 40
exit_group(1)   = ?
+++ exited with 1 +++
moder8@telecom:~/bin$

As you can see, the file "entities" shows up in an "ls" listing, but not 
when I try to run it by itself or with strace.


Thanks for your help!

--
E. William Horne
William Warren Consulting
339-364-8487

___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Joe Polcari]

It appears you're in a different directory. Try:
strace /home/moder8/bin/entities

-Original Message-
From: discuss-bounces+joe=polcari@blu.org
[mailto:discuss-bounces+joe=polcari@blu.org] On Behalf Of Bill Horne
Sent: Tuesday, July 29, 2014 3:41 PM
To: BLU Discussion List
Subject: Re: [Discuss] Seeking information on binaries called "entities" and
"fixup"

On 7/28/2014 5:33 PM, Tom Metro wrote:
> Bill Horne wrote:
>> No, they work on the old server, but fail on the new. I assume it's a
>> permissions issue, but I can't figure out what might cause it.
> % strace entities

Tom,

Thanks for the suggestion. Here's a log snippet that may make this issue 
more clear:

moder8@telecom:~/bin$ ls -lh /home/moder8/bin/entities
-rwxrwxr-x 1 moder8 telecom 8.8K Jan 27  2012 /home/moder8/bin/entities

moder8@telecom:~/bin$ ls -lh 
/var/www/html/archives/back.issues/recent.single.issues/I125
-rw-rw-r-- 1 root root 3.1K Jul 28 11:12 
/var/www/html/archives/back.issues/recent.single.issues/I125

moder8@telecom:~/bin$ strace ./entities 
/var/www/html/archives/back.issues/recent.single.issues/I125
execve("./entities", ["./entities", 
"/var/www/html/archives/back.issu"...], [/* 22 vars */]) = -1 ENOENT (No 
such file or directory)
write(2, "strace: exec: No such file or di"..., 40strace: exec: No such 
file or directory
) = 40
exit_group(1)   = ?
+++ exited with 1 +++
moder8@telecom:~/bin$

As you can see, the file "entities" shows up in an "ls" listing, but not 
when I try to run it by itself or with strace.

Thanks for your help!

-- 
E. William Horne
William Warren Consulting
339-364-8487

___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Bill Horne]

On 7/29/2014 3:44 PM, Joe Polcari wrote:

strace /home/moder8/bin/entities

Joe,

Thanks for your suggestion: here's the output.

moder8@telecom:~$ strace /home/moder8/bin/entities 
/var/www/html/archives/back.issues/recent.single.issues/I125
execve("/home/moder8/bin/entities", ["/home/moder8/bin/entities", 
"/var/www/html/archives/back.issu"...], [/* 21 vars */]) = -1 ENOENT (No 
such file or directory)
write(2, "strace: exec: No such file or di"..., 40strace: exec: No such 
file or directory

) = 40
exit_group(1)   = ?
+++ exited with 1 +++
moder8@telecom:~$ pwd
/home/moder8

Bill

--
E. William Horne
William Warren Consulting
339-364-8487

___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Tom Metro]

Bill Horne wrote:
> moder8@telecom:~/bin$ ls -lh /home/moder8/bin/entities
> -rwxrwxr-x 1 moder8 telecom 8.8K Jan 27  2012 /home/moder8/bin/entities
> 
> moder8@telecom:~$ strace /home/moder8/bin/entities 
> /var/www/html/archives/back.issues/recent.single.issues/I125
> execve("/home/moder8/bin/entities", ["/home/moder8/bin/entities", 
> "/var/www/html/archives/back.issu"...], [/* 21 vars */]) = -1 ENOENT (No such 
> file or directory)
> write(2, "strace: exec: No such file or di"..., 40strace: exec: No such file 
> or directory
> ) = 40

Have you examined /home/moder8/bin/entities with 'file', strings, and less?

I would guess that it is a shell script with a missing interpreter, but
the error message is not right for that.

Actually, that might be it:

% touch foo
% chmod u+x foo
% echo #\!/bin/bogus > foo
fringe:/tmp% strace ./foo
execve("./foo", ["./foo"], [/* 54 vars */]) = -1 ENOENT (No such file or
directory)
[...]
write(3, "strace: exec: No such file or di"..., 40strace: exec: No such
file or directory
) = 40

It's a misleading error message. The "No such file" is not referring to
./foo, but to the specified interpreter, /bin/bogus, but because the
bang-path magic is embedded in execve(), and it only returns an error
code (it doesn't generate the error message to STDERR), you're left with
a generic error and no object being identified.


I assume the above was on the new server. What happens when you strace
it on the old server?

 -Tom

-- 
Tom Metro
The Perl Shop, Newton, MA, USA
"Predictable On-demand Perl Consulting."
http://www.theperlshop.com/
___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Greg Rundlett (freephile)]

On Sun, Jul 27, 2014 at 11:08 PM, Bill Horne  wrote:

> I'm moving the Telecom Digest to a new server, and I'm seeking information
> about a binary named "entities", and one named "fixup".  I've found them in
> a script that processes emails into html pages for publications, but the
> script's author isn't available, and neither is working.
>
> AFAICT, "entities" is supposed to replace ASCII characters with HTML
> Entities where needed, and I don't know what "fixup" is supposed to do.
>
>

[replying to all this time]

Here is a PHP function that could help you replace the 'entities' binary

/**
 * Replaces ampersands with html entity, making it safe for XML
 * use lookahead assertion that the ampersand is not followed by an
optional hash, one or more
 * word characters and a semicolon.  In other words, it WILL match if it's
a bare ampersand,
 * such as in the string "H&R Block" but
 * will leave friendly entities like " or Õ alone
 * @see http://www.php.net/manual/en/regexp.reference.assertions.php
 * @param string $message
 * @return string
 */
function entitySafe($message)
{
return preg_replace('/&(?!#?\w+;)/', '&', $message);
}

Greg Rundlett
http://eQuality-Tech.com 
http://freephile.org
___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Bill Horne]

On 7/29/2014 6:06 PM, Tom Metro wrote:

Bill Horne wrote:

moder8@telecom:~/bin$ ls -lh /home/moder8/bin/entities
-rwxrwxr-x 1 moder8 telecom 8.8K Jan 27  2012 /home/moder8/bin/entities

moder8@telecom:~$ strace /home/moder8/bin/entities 
/var/www/html/archives/back.issues/recent.single.issues/I125
execve("/home/moder8/bin/entities", ["/home/moder8/bin/entities", 
"/var/www/html/archives/back.issu"...], [/* 21 vars */]) = -1 ENOENT (No such file or directory)
write(2, "strace: exec: No such file or di"..., 40strace: exec: No such file or 
directory
) = 40

Have you examined /home/moder8/bin/entities with 'file', strings, and less?

I would guess that it is a shell script with a missing interpreter, but
the error message is not right for that.

Actually, that might be it:

% touch foo
% chmod u+x foo
% echo #\!/bin/bogus>  foo
fringe:/tmp% strace ./foo
execve("./foo", ["./foo"], [/* 54 vars */]) = -1 ENOENT (No such file or
directory)
[...]
write(3, "strace: exec: No such file or di"..., 40strace: exec: No such
file or directory
) = 40

It's a misleading error message. The "No such file" is not referring to
./foo, but to the specified interpreter, /bin/bogus, but because the
bang-path magic is embedded in execve(), and it only returns an error
code (it doesn't generate the error message to STDERR), you're left with
a generic error and no object being identified.


I assume the above was on the new server. What happens when you strace
it on the old server?

  -Tom



Tom,

I ran the "entities" binary on the "old" machine, and I've pasted the 
output here: "I125" is a standard MBOX-format email file, with a single 
issue of The Telecom Digest in it. It works as expected.


moder8@old-massis:/tmp$ strace -o/tmp/strace_out2.txt 
/home/moder8/bin/entities 

execve("/home/moder8/bin/entities", ["/home/moder8/bin/entities"], [/* 
20 vars */]) = 0

brk(0)  = 0x8749000
access("/etc/ld.so.nohwcap", F_OK)  = -1 ENOENT (No such file or 
directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb77df000
access("/etc/ld.so.preload", R_OK)  = -1 ENOENT (No such file or 
directory)

open("/etc/ld.so.cache", O_RDONLY)  = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=23117, ...}) = 0
mmap2(NULL, 23117, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb77d9000
close(3)= 0
access("/etc/ld.so.nohwcap", F_OK)  = -1 ENOENT (No such file or 
directory)

open("/lib/i686/cmov/libc.so.6", O_RDONLY) = 3
read(3, 
"\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0n\1\0004\0\0\0"..., 
512) = 512

fstat64(3, {st_mode=S_IFREG|0755, st_size=1327556, ...}) = 0
mmap2(NULL, 1337704, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 
0) = 0xb7692000

mprotect(0xb77d2000, 4096, PROT_NONE)   = 0
mmap2(0xb77d3000, 12288, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x140) = 0xb77d3000
mmap2(0xb77d6000, 10600, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb77d6000

close(3)= 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb7691000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb76916c0, 
limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, 
limit_in_pages:1, seg_not_present:0, useable:1}) = 0

mprotect(0xb77d3000, 8192, PROT_READ)   = 0
mprotect(0xb77fd000, 4096, PROT_READ)   = 0
munmap(0xb77d9000, 23117)   = 0
fstat64(0, {st_mode=S_IFREG|0666, st_size=3174, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb77de000

read(0, "From telecom-owner+DV033N00125=b"..., 4096) = 3174
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb77dd000

write(1, "From telecom-owner+DV033N00125=b"..., 108) = 108
write(1, "Return-path: 

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Greg Rundlett (freephile)]

>
> On 7/29/2014 6:06 PM, Tom Metro wrote:
>>
>> Have you examined /home/moder8/bin/entities with 'file', strings, and
>> less?
>>
>> I would guess that it is a shell script with a missing interpreter,
>>
>
As Tom suggested, what do you get with
strings entities
and
file entitites

Greg Rundlett
http://eQuality-Tech.com 
http://freephile.org
___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Bill Horne]

On Fri, 1 Aug 2014 01:16:33 -0400 
"Greg Rundlett (freephile)"  wrote:

> As Tom suggested, what do you get with
> strings entities
> and
> file entitites

This is the output of the "strings"command on the "old" machine

moder8@old-massis:~/rsi$ strings /home/moder8/bin/entities http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Bill Horne]

On Fri, 1 Aug 2014 01:16:33 -0400 
"Greg Rundlett (freephile)"  wrote:

> As Tom suggested, what do you get with
> strings entities
> and
> file entitites

This is the output of the "strings"command on the "old" machine

moder8@old-massis:~/rsi$ strings /home/moder8/bin/entities http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Tom Metro]

Bill Horne wrote:
> moder8@old-massis:~/rsi$ strings /home/moder8/bin/entities  @(#) $Id: entities.c,v 1.4 2011/12/19 12:40:17 moder8 Exp $
[...]
> Now, the output of the "files" command on the old machine
> 
> /home/moder8/bin/entities: ELF 32-bit LSB executable, Intel 80386,\
> version 1 (SYSV), dynamically linked (uses shared libs),\
>  for GNU/Linux 2.6.8, not stripped

So clearly you're dealing with a compiled binary. The RCS comment line
captured by strings suggests C code.

It could be failing to run on the new machine due to an architecture
mismatch (different CPU family, 64-bit vs. 32-bit), or due to missing
dynamically linked libraries:

> /lib/ld-linux.so.2
> libc.so.6

(Though the error message is usually more helpful in either of those cases.)

One answer is to rebuild entities.c, if you have the source, but if you
did, you probably wouldn't have started this thread.

Some trial-and-error might get them running on the new machine by
pulling in the necessary dynamic libraries, but that's not a great
solution. It just defers the headache until your next server move.


Quoting your original message:
> ...I'm seeking information about a binary named "entities", and one
> named "fixup".  I've found them in a script that processes emails
> into html pages for publications, but the script's author isn't
> available, and neither is working.

I gather the script is custom and not from some project? My guess would
be that these tools were borrowed from some other project. Probably a
mail archiving tool. Some more searching might turn up their origin.

What's the big picture that you are trying to accomplish with the
script? Presenting the Telecom Digest on a web site? Maybe an
off-the-shelf mail archiving tool is a better way to go? (Like MHonArc.)

What language is the script written in? 'fixup' remains a mystery, but
you probably know enough about 'entities' to replace it with some
in-line code. Greg gave you a PHP example. It could be done as a
one-liner in Perl, with the assistance of a module.

 -Tom

-- 
Tom Metro
The Perl Shop, Newton, MA, USA
"Predictable On-demand Perl Consulting."
http://www.theperlshop.com/
___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Greg Rundlett (freephile)]

In the 'strings' output, I noticed a typo - perfect for searching code
because you hope people don't make too many of the SAME typos across
multiple projects.

What I noticed is that one of the entities was &pount; (instead of £)
 When searching OHLOH (now Black Duck), it found 177 matches.  Sadly their
engine does not search literal strings even when they say they do... they
ignore the & and the ;  Still, it's a really good start at a very small
haystack for finding the original source of this code.
https://code.ohloh.net/search?s=%26pount

Then I added the term brvbar in a compound search, but that turned up zero
results - which means that this original source is not found in their
database.


On Sat, Aug 2, 2014 at 1:42 AM, Tom Metro  wrote:

> What's the big picture that you are trying to accomplish with the
> script? Presenting the Telecom Digest on a web site? Maybe an
> off-the-shelf mail archiving tool is a better way to go? (Like MHonArc.)
>

Agreed, if you just want to put the Digest on the web, then I'd feed it to
MHonArc, or similar.

Out of curiosity, I looked at the MHonArc sources to see if there were such
files (entities and fixup), but no.  MHonArc is written in Perl.  That
said, there is quite a bit of code to deal with sanitizing or escaping
content for conversion to HTML for display.  To wit:
http://www.mhonarc.org/release/MHonArc/latest/lib/mhtxtplain.pl
http://www.mhonarc.org/release/MHonArc/latest/lib/mhtxtenrich.pl
http://www.mhonarc.org/release/MHonArc/latest/lib/mhtxthtml.pl

And, even with that, there is a big security warning on the homepage for
yet another script injection vulnerability.  That's because MHonArc doesn't
fully parse the HTML, nor does it use a white-listing approach.  Basically,
with MHonArc, you should not enable HTML in your mail archives.  Despite
the obvious "weakness" of MHonArc, I'm simply pointing out the obvious that
rolling your own solution that mimics MHonArc's feature set would be quite
a project.


>
> What language is the script written in? 'fixup' remains a mystery,
>

Fixup is a mystery, but at least one thing it probably did was deal with
angle brackets and keywords.


> but
> you probably know enough about 'entities' to replace it with some
> in-line code. Greg gave you a PHP example. It could be done as a
> one-liner in Perl, with the assistance of a module.
>



Greg Rundlett
http://eQuality-Tech.com
http://freephile.org
___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss

Re: [Discuss] Seeking information on binaries called "entities" and "fixup"

[Bill Horne]

On 8/2/2014 1:42 AM, Tom Metro wrote:

Quoting your original message:

...I'm seeking information about a binary named "entities", and one
named "fixup".  I've found them in a script that processes emails
into html pages for publications, but the script's author isn't
available, and neither is working.

I gather the script is custom and not from some project? My guess would
be that these tools were borrowed from some other project. Probably a
mail archiving tool. Some more searching might turn up their origin.


I didn't see any mention of the name anywhere in this context.


What's the big picture that you are trying to accomplish with the
script? Presenting the Telecom Digest on a web site? Maybe an
off-the-shelf mail archiving tool is a better way to go? (Like MHonArc.)


AFAICT, it's intended to replace HTML reserved characters with HTML 
entities. Less-than becomes "<", etc.



What language is the script written in? 'fixup' remains a mystery, but
you probably know enough about 'entities' to replace it with some
in-line code. Greg gave you a PHP example. It could be done as a
one-liner in Perl, with the assistance of a module.


I don't know what language it was written in. I accepted help from a 
volunteer who installed several scripts, and this binary was in one of them.


I'll try the PHP that Greg suggested, and rebuild the process.

Bill

--
E. William Horne
William Warren Consulting
339-364-8487

___
Discuss mailing list
Discuss@blu.org
http://lists.blu.org/mailman/listinfo/discuss