phaqphaq

Today a colleague of mine faced a very weird problem.
While parsing XML output from an HP ILO into Perl, his code constantly broke with the message:

FILE.XML:123 parser error : PCDATA invalid Char value 1

While the message itself states clearly that there is an unexpected character value (Char value 1, ASCII SOH) on one hand, it doesn’t tell the character position on the other.

Looking at the input string itself on the console, it wasn’t obvious either:

<EVENT SEVERITY=”Caution” LAST_UPDATE=”08/03/2009 22:20″ INITIAL_UPDATE=”08/03/2009 22:20″ COUNT=”1″ DESCRIPTION=”POST Error: ” />

So I proposed to add some lines to help identify the character position on the given input string, which was basically this:

@array = unpack(”C*”, $_my_input_var);
foreach (@array) {
printf(”char \”%s\” is ord %s\n”, chr($_), $_);
}

This led to the following output:

char “<" is ord 60
char "E" is ord 69
char "V" is ord 86
--- some output omitted ---
char ":" is ord 58
char " " is ord 32
char "" is ord 1
char """ is ord 34
char " " is ord 32
char "/" is ord 47
char ">” is ord 62

So looking at this we saw that ASCII char 1 (which is an unprintable character, it will be represented as ^A in some editors like vi) was the fifth character before the end of the string.

Well, basically the solution to this is to apply an additional input filter to remove ASCII char 1 like this:

$_my_input_var =~ s/\x01//g;

While this solves just _this_ problem, a more solid solution is to remove all non-printable characters as well, given the list of ASCII characters at http://www.asciitable.com/.

So a filter like this may apply, removing all non-printable characters, leaving just a few control characters 1×08 to 1×1F (Tab, Carriage Return, Line Feed and a few others) and the printable characters in it.

$_my_input_var =~ s/[\x00-\x08\x0B-\x1F\x7F-\xFF]//g;

One might asume building a port from source should be fast and straight forward.
Not in this case, however…

# cd /usr/ports/devel/p5-File-Tail
# make clean build
===> Cleaning for perl-5.8.8
===> Cleaning for p5-File-Tail-0.99.3
===> Vulnerability check disabled, database not found
===> Extracting for p5-File-Tail-0.99.3
=> MD5 Checksum OK for File-Tail-0.99.3.tar.gz.
=> SHA256 Checksum OK for File-Tail-0.99.3.tar.gz.
===> p5-File-Tail-0.99.3 depends on file: /usr/local/bin/perl5.8.8 – found
===> Patching for p5-File-Tail-0.99.3
===> p5-File-Tail-0.99.3 depends on file: /usr/local/bin/perl5.8.8 – found
===> Applying FreeBSD patches for p5-File-Tail-0.99.3
===> p5-File-Tail-0.99.3 depends on file: /usr/local/bin/perl5.8.8 – found
===> Configuring for p5-File-Tail-0.99.3

File::Tail will be installed without debugging information.
This information isn’t usefull unless you intend to tinker
with the code. To install with debugging enabled, use:
perl Makefile.PL LOGIT
Checking if your kit is complete…
Looks good
Writing Makefile for Mail
==> Your Makefile has been rebuilt. <==
==> Please rerun the make command. <==
false
*** Error code 1

Stop in /usr/ports/devel/p5-File-Tail/work/File-Tail-0.99.3

Usually, this particular error happens only if local time/date is not set correctly.

Good, so show me the date, please:

# date
Wed Jan 10 11:49:18 CET 2007

But what is this? Date and time seems right.
Maybe something wrong with the files within the port directory?

# cd /usr/src/devel/p5-File-Tail
# make clean && make extract
# find ./work/File-Tail-0.99.3/ -type f -exec ls -l {} \;
-rw-r--r-- 1 root wheel 452 Sep 14 2005 ./META.yml
-rw-r--r-- 1 root wheel 1931 Dec 18 1998 ./t/20tail.t
-rw-r--r-- 1 root wheel 912 Sep 14 2005 ./t/30name_change.t
-rw-r--r-- 1 root wheel 1653 Dec 18 1998 ./t/10open.t
-rwxr-xr-x 1 root wheel 1101 Feb 2 1999 ./select_demo
-rw-r--r-- 1 root wheel 27994 Jan 10 11:47 ./Tail.pm.debug
-rw-r--r-- 1 root wheel 4812 Sep 14 2005 ./Changes
-rwxr-xr-x 1 root wheel 1381 Nov 3 1999 ./logwatch
-rw-r--r-- 1 root wheel 198 Sep 14 2005 ./MANIFEST
-rw-r--r-- 1 root wheel 25311 Jan 10 11:47 ./Tail.pm
-rw-r--r-- 1 root wheel 2471 Oct 17 2000 ./Makefile.PL
-rw-r--r-- 1 root wheel 2143 Feb 2 1999 ./README
-rw-r--r-- 1 root wheel 27991 Sep 14 2005 ./Tail.pm.debug.orig
-rw-r--r-- 1 root wheel 23027 Jan 10 11:47 ./Makefile
-rw-r--r-- 1 root wheel 0 Jan 10 11:47 ./blib/lib/File/.exists
-r--r--r-- 1 root wheel 25311 Feb 9 11:47 ./blib/lib/File/Tail.pm
-rw-r--r-- 1 root wheel 0 Jan 10 11:47 ./blib/lib/auto/File/Tail/.exists
-rw-r--r-- 1 root wheel 0 Jan 10 11:47 ./blib/arch/.exists
-rw-r--r-- 1 root wheel 0 Jan 10 11:47 ./blib/arch/auto/File/Tail/.exists
-rw-r--r-- 1 root wheel 0 Jan 10 11:47 ./blib/bin/.exists
-rw-r--r-- 1 root wheel 0 Jan 10 11:47 ./blib/script/.exists
-rw-r--r-- 1 root wheel 0 Jan 10 11:47 ./blib/man1/.exists
-rw-r--r-- 1 root wheel 0 Jan 10 11:47 ./blib/man3/.exists
-rw-r--r-- 1 root wheel 14740 Jan 10 11:47 ./blib/man3/File::Tail.3
-rw-r--r-- 1 root wheel 0 Jan 10 11:47 ./pm_to_blib

Well, nothing bad here either.
However, it *has* to be something about the date.

Maybe, there is some issue with perl libs, MakeMaker in
particular, then?

Let's see:

# find /usr/local/lib/perl5/ -name "*MakeMaker*" -exec ls -l {} \;
-r--r--r-- 1 root wheel 78826 Feb 9 08:48 /usr/local/lib/perl5/5.8.8/ExtUtils/MakeMaker.pm
total 20
-r--r--r-- 1 root wheel 565 Feb 9 08:48 Config.pm
-r--r--r-- 1 root wheel 7605 Feb 9 08:48 FAQ.pod
-r--r--r-- 1 root wheel 4387 Feb 9 08:48 Tutorial.pod
-r--r--r-- 1 root wheel 769 Feb 9 08:48 bytes.pm
-r--r--r-- 1 root wheel 707 Feb 9 08:48 vmsish.pm
-rw-r--r-- 1 root wheel 19874 Feb 9 08:49 /usr/local/lib/perl5/5.8.8/perl/man/man3/ExtUtils::MakeMaker.3.gz
-rw-r--r-- 1 root wheel 1866 Feb 9 08:49 /usr/local/lib/perl5/5.8.8/perl/man/man3/ExtUtils::MakeMaker::Config.3.gz
-rw-r--r-- 1 root wheel 1983 Feb 9 08:49 /usr/local/lib/perl5/5.8.8/perl/man/man3/ExtUtils::MakeMaker::bytes.3.gz
-rw-r--r-- 1 root wheel 3631 Feb 9 08:49 /usr/local/lib/perl5/5.8.8/perl/man/man3/ExtUtils::MakeMaker::Tutorial.3.gz
-rw-r--r-- 1 root wheel 5239 Feb 9 08:49 /usr/local/lib/perl5/5.8.8/perl/man/man3/ExtUtils::MakeMaker::FAQ.3.gz
-rw-r--r-- 1 root wheel 1937 Feb 9 08:49 /usr/local/lib/perl5/5.8.8/perl/man/man3/ExtUtils::MakeMaker::vmsish.3.gz
-r--r--r-- 1 root wheel 3905 Feb 9 08:49 /usr/local/lib/perl5/5.8.8/BSDPAN/ExtUtils/MakeMaker.pm

There it is! You see?
Obviously the was a timestep forward one the system clock sometimes in the past, so files created then actually got a timestamp in the future.

So let's fix this and try rebuilding the port.

# find /usr/local/lib/perl5/ -name "*MakeMaker*" -exec touch {} \;
# cd /usr/ports/devel/p5-File-Tail
# make clean build
===> Cleaning for perl-5.8.8
===> Cleaning for p5-File-Tail-0.99.3
===> Vulnerability check disabled, database not found
===> Extracting for p5-File-Tail-0.99.3
=> MD5 Checksum OK for File-Tail-0.99.3.tar.gz.
=> SHA256 Checksum OK for File-Tail-0.99.3.tar.gz.
===> p5-File-Tail-0.99.3 depends on file: /usr/local/bin/perl5.8.8 – found
===> Patching for p5-File-Tail-0.99.3
===> p5-File-Tail-0.99.3 depends on file: /usr/local/bin/perl5.8.8 – found
===> Applying FreeBSD patches for p5-File-Tail-0.99.3
===> p5-File-Tail-0.99.3 depends on file: /usr/local/bin/perl5.8.8 – found
===> Configuring for p5-File-Tail-0.99.3

File::Tail will be installed without debugging information.
This information isn’t usefull unless you intend to tinker
with the code. To install with debugging enabled, use:
perl Makefile.PL LOGIT
Checking if your kit is complete…
Looks good
Writing Makefile for File::Tail
===> Building for p5-File-Tail-0.99.3
cp Tail.pm blib/lib/File/Tail.pm
Manifying blib/man3/File::Tail.3

So we learn an important lesson from this: Perl’s MakeMaker does not only fail the build process if the system clock is set incorrectly.
It does so also if the timestamps on any source files and MakeMaker’s own perl modules have a time leap forward.

Well, today I just did not believe my eyes.

I was just bringing up a new webmail host when I noticed Apache throwing an 500 internal server error at me.

Looking at the error log I saw this line:

[Wed Jan 01 00:16:35 2003] [error] [client 127.0.0.1] Prototype mismatch: sub APR::XSLoader::BOOTSTRAP: none vs () at /usr/local/lib/perl5/site_perl/5.8.8/mach/APR/XSLoader.pm line 24. BEGIN failed–compilation aborted at /usr/local/lib/perl5/site_perl/5.8.8/mach/APR/XSLoader.pm line 26. Compilation failed in require at /usr/local/lib/perl5/site_perl/5.8.8/mach/Apache2/Reload.pm line 161.

At first I though to be going mad because that server had just worked only few hours ago. Why would it fail like that after relocating it?

At second thought I noticed the time stamp which said “Wed Jan 01 00:16:35 2003″.

Well, the server had likely lost it’s CMOS settings during relocation. So after correcting the time through ntpdate the error had disappeared and Apache/mod_perl worked like a charm again.

Imagine you have a directory with thousands of text files, each of which containing some keywords to be replaced.

While there’s a lot of tools out there allowing you to find and replace strings in text files, there’s always a goog reason to use the tools that you already have.

This snippet uses find in combination with perl to achieve the task:

#find /my/path -type f -exec perl -i -p -e ’s/search/replace/g;’ {} \;

This command line instructs to find all files within /my/path, passing them to perl, which is instructed to find and replace given strings instantly.

Rememver however that this is not binary sage so you should run this only on directories containt text files.

The get additional verbosity add the ‘-print’ flag to find’s command line to see which files is being worked on.

If you pass an optional extension to ‘-i’ (eg. ‘-i.bak’) perl is instructed to create a backup by the same filename plus extension.

As always: man find and man perl are your friends.

Enjoy!

When I was setting up a development box today I encountered a strange error while compiling the Mail::ClamAV perl module.

[output stripped]
Starting Build Compile Stage
Starting “perl Makefile.PL” Stage
Note (probably harmless): No library found for -lclamav
Writing Makefile for Mail::ClamAV
Finished “perl Makefile.PL” Stage
Starting “make” Stage
/usr/local/bin/perl /usr/local/lib/perl5/5.8.8/ExtUtils/xsubpp -typemap /usr/local/lib/perl5/5.8.8/ExtUtils/typemap ClamAV.xs > ClamAV.xsc && mv ClamAV.xsc ClamAV.c
cc -c -I/usr/ports/mail/p5-Mail-ClamAV/work/Mail-ClamAV-0.17 -DAPPLLIB_EXP=”/usr/local/lib/perl5/5.8.8/BSDPAN” -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -Wdeclaration-after-statement -g -DVERSION=\”0.17\” -DXS_VERSION=\”0.17\” -DPIC -fPIC “-I/usr/local/lib/perl5/5.8.8/mach/CORE” ClamAV.c
ClamAV.xs:11:20: clamav.h: No such file or directory
ClamAV.xs:19: error: field `limits’ has incomplete type
ClamAV.xs:20: error: field `st’ has incomplete type
ClamAV.xs: In function `clamav_perl_new’:
ClamAV.xs:48: error: invalid application of `sizeof’ to incomplete type `cl_stat’
[output stripped]

At first I thought of a bug in the module package itself, but after further inspection I noticed these two lines:

>> Note (probably harmless): No library found for -lclamav
>> ClamAV.xs:11:20: clamav.h: No such file or directory

So at a first glance it would look like libclamav and it’s header file were not installed properly (hence not found), but since it did so only minutes ago there must have been another reason for this error to come up.

Looking at the cc command line I noticed the absence of the usual include statement pointing to /usr/local/include.

To fix this issue we need to know how module compilation actually works in Perl.

Some may be familiar with autoconfig (configure.sh) used with most source tarballs out there. Perl uses a similar approach to bring together it’s build dependencies called MakeMaker (ExtUtils::MakeMaker). Perl module packages usually come along with a MakeMaker prototype file (Makefile.PL) which serves the purpose to build a regurlar Makefile.

MakeMaker relies on Perl’s bootstrap build settings to stick things together, so if anything is missing there errors at compile time may arise.

So we need to check out the Perl configuration files Config.pm and Config_heavy.pl. These are usually found at /usr/local/lib/perl5/x.y.z/mach (x.y.z denotes the Perl version, e.g. 5.8.8. If this does not apply to you, check your ‘perl -V’ output and look at your @INC directories, also [s]locate/find may be of use too).

Let’s look at Config.pm first, locating this line:

libpth=’/usr/lib’,

Change it to to include “/usr/local/lib” (or any lib directory that may apply to your environment) and save the file.

libpth=’/usr/lib /usr/local/lib’,

This will cause MakeMaker to not claim any longer about “No library found for -lclamav”.

Doing the same for the header files does involve a bit of additional work. Let’s check out Config_heavy.pl.
This file contains a lot of settings written out by autoconf during Perl’s initial bootstrap compilation, some of which are used to compile inline code in Perl.

Let’s look at the ccflags/cppflags first:

ccflags=’-DAPPLLIB_EXP=”/usr/local/lib/perl5/5.8.8/BSDPAN” -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe’
cppflags=’-DAPPLLIB_EXP=”/usr/local/lib/perl5/5.8.8/BSDPAN” -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe’

Adding the include directory (/usr/local/include or whatever may apply to you) will cause the header files to be found at compile time:

ccflags=’-DAPPLLIB_EXP=”/usr/local/lib/perl5/5.8.8/BSDPAN” -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -I/usr/local/include’
cppflags=’-DAPPLLIB_EXP=”/usr/local/lib/perl5/5.8.8/BSDPAN” -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -I/usr/local/include’

Since compiling usually also involves linking these additional settings must be changed to:

lddlflags=’-shared ‘
ldflags=’ -Wl,-E’
libpth=’/usr/lib’
libspath=’ /usr/lib’

Change the lines to include the library directory (/usr/local/lib in my case) to ensure ld finding all dependencies:

lddlflags=’-shared -L/usr/local/lib’
ldflags=’ -Wl,-E -L/usr/local/lib’
libpth=’/usr/lib /usr/local/lib’
libspath=’ /usr/lib /usr/local/lib’

After applying all these changes I could finally compile and install Mail::ClamAV with success.

After all only one question remains: Why was my Perl config so totally screwed up? But this is another story…