Perl Module Mechanics

This page describes the mechanics of creating, compiling, testing, releasing, and maintaining Perl modules.

This is not a reference manual. Rather, it is a running account of how to do these things. More to the point, it is an account of how I do these things. Accordingly

Perl modules

A Perl module is a self-contained piece of Perl code that can be used by a Perl program or by other Perl modules. It is conceptually similar to a C link library, or a C++ class.

CPAN

The Comprehensive Perl Archive Network (CPAN) contains a huge collection of publicly available modules, and includes guidelines for module creation and use. This document provides introduction and overview; CPAN should be consulted for detail and references.

Module names

Each Perl module has a name. Module names must be unique. To minimize namespace collisions, Perl provides a hierarchal namespace for modules, similar to the namespace for Java classes.

Components of a module name are separated by double colons (::). Thus, we can have

Module Function
Math::Complex Complex number data type
Math::Approx Approximate x,y-values by a function
String::Approx Approximate string matching and substitution
String::BitCountCount number of "1" bits in strings

without conflict.

One component, two component, three component, four...

There is no restriction on the number of components in a module name; however, most ordinary application modules have two-component names.

One-component module names are a limited resource, and should only be taken for patently global things:

ModuleFunction
ConfigStores details of the Perl build configuration

Names with three or more components risk confusing users:

Will the real module please stand up?

Module names vs. Inheritance

The module name hierarchy doesn't have anything to do with class inheritance. Math::Complex might be a subclass of Math, or it might not. It might be a subclass of String::Approx, for that matter.

Module files

Each module is contained in a single file. Module files are stored in a subdirectory hierarchy that parallels the module name hierarchy. Thus,

Module Is Stored In
Config Config.pm
Math::Complex Math/Complex.pm
String::ApproxString/Approx.pm

All module files have an extension of .pm. pm stands for Perl Module.

You may also encounter .pl files. pl stands for Perl Library. .pl files were used in Perl4; .pm files are preferred in Perl5.

Module libraries

The Perl interpreter has a list of directories where it searches for modules. Within a Perl program, this list is available in the global array @INC. INC is short for "include".

To see the initial contents of @INC, do 

>perl -V
 ...
@INC: 
/usr/local/lib/perl5/5.8.6/i686-linux
/usr/local/lib/perl5/5.8.6
/usr/local/lib/perl5/site_perl/5.8.6/i686-linux
/usr/local/lib/perl5/site_perl/5.8.6
/usr/local/lib/perl5/site_perl/5.8.5/i686-linux
/usr/local/lib/perl5/site_perl/5.8.5
/usr/local/lib/perl5/site_perl
.
As installed on my machine, the Perl interpreter looks for modules in eight places. Three of these have i686-linux in their path; these are for modules that link to separately compiled C code, and are therefore architecture dependent. The last is the current working directory.

/usr/local/lib/perl5/5.8.6 contains modules that come with the standard Perl5 distribution. /usr/local/lib/perl5/5.8.5 contains modules that came with the previous version of Perl. Modules from successive distributions are kept in different directories to support programs that may rely on older versions.

Finally, /usr/local/lib/perl5/site_perl/ contains modules that are built and installed locally; i.e. are not part of the standard distribution. Local and standard modules are separated in this way so that a new release of the interpreter can replace the standard modules without disturbing the locally installed ones.

So, if the modules listed above were installed on my system, they would be in

Module Path
Config /usr/local/lib/perl5/site_perl/5.8.6/Config.pm
Math::Complex /usr/local/lib/perl5/site_perl/5.8.6/Math/Complex.pm
String::Approx /usr/local/lib/perl5/site_perl/5.8.6/String/Approx.pm

Ready, Set, Go

Perl is an interpreted language, and has not the source code/object code distinction that exists in languages like C or Java. The programmer writes a .pm file, and the interpreter executes it. This means that you can edit the .pm files in /usr/local/lib/perl5/site_perl/, and your edits will be effective immediately.

Don't.

Many people may be running programs that depend on the modules in /usr/local/lib/perl5/site_perl. When the modules break, the programs break, and then there are users lined up at your door.

Development directories

You need to establish a development area for developing Perl modules. It could be in /usr/local/src, it could be in your home directory, or it could be somewhere else. It should not be in /usr/local/lib/perl5/site_perl/.

I'll write .../development/ to refer to the development directory, where ever it is.

How hard could it be?

Building and installing Perl modules would seem simple. They aren't compiled or linked. All you have to do is copy the .pm file from the development area to /usr/local/lib/perl5.

For ordinary Perl modules, this is true. However. Some Perl modules link to dynamically loaded shared C libraries, via a special interface language called XS. This not only requires compiling and linking, but potentially involves all the worst kinds of system dependencies.

h2xs

To get you started on this perilous journey, Perl provides a utility called h2xs. h2xs reads the .h file for a C library and creates a skeleton for the .xs file that will be required to build a Perl module that links to that library. It also constructs a fairly elaborate system of directories and makefiles in which to build and test the Perl module.

Over time, the structure that h2xs creates has become the standard for building Perl modules—even modules that don't link to C code. I use this structure, because

Creating a Perl Module

To create a new perl module, go to your development area and do 
.../development>h2xs -X -n Foo::Bar
Writing Foo-Bar/lib/Foo/Bar.pm
Writing Foo-Bar/Makefile.PL
Writing Foo-Bar/README
Writing Foo-Bar/t/Foo-Bar.t
Writing Foo-Bar/Changes
Writing Foo-Bar/MANIFEST
.../development>
The -X switch specifies that this module does not link to C code. The -n switch gives the name of the new module.

h2xs creates Foo-Bar/ and populates it with the six files listed. Foo-Bar/ is called the module directory or the build directory.

Let's look at the six files.

MANIFEST

MANIFEST lists the files in the build directory. MakeMaker checks that all the files listed in MANIFEST are present, and complains if they aren't. This is mainly used to ensure that module distributions are received intact.

README

README contains a skeleton for a README file. Creating a README file is optional, but considered good practice. Modules that are uploaded to CPAN are required to have README files.

The README file also contains this boilerplate 

Copyright (C) 2005 your name here

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself, either Perl version 5.8.6 or,
at your option, any later version of Perl 5 you may have available.
Perl is distributed under the terms of the GPL and the Artistic License; if these aren't the terms that you want for your module, you need to change this language.

Changes

This is a file to record a change history of the module. h2xs creates it, but it's up to the programmer to maintain it.

t/Foo-Bar.t

t/ is a subdirectory for test files, and t/Foo-Bar.t contains a skeleton test routine for the module. As created, all it does is verify that the module can be loaded into a Perl program. If nothing else, this will catch syntax errors in the module. Programmers are encouraged to create functional tests for their modules.

Makefile.PL

A single makefile could not manage all the system configuration issues involved in building a Perl XS module. Rather than try, h2xs creates Makefile.PL. Makefile.PL. contains a short Perl program: 
.../development/Foo-Bar>cat Makefile.PL
use 5.008006;
use ExtUtils::MakeMaker;
# See lib/ExtUtils/MakeMaker.pm for details of how to influence
# the contents of the Makefile that is written.
WriteMakefile(
    NAME              => 'Foo::Bar',
    VERSION_FROM      => 'lib/Foo/Bar.pm', # finds $VERSION
    PREREQ_PM         => {}, # e.g., Module::Name => 1.1
    ($] >= 5.005 ?     ## Add these new keywords supported since 5.005
      (ABSTRACT_FROM  => 'lib/Foo/Bar.pm', # retrieve abstract from module
       AUTHOR         => 'your name <your email address>') : ()),
);
The crucial line is the second one: use ExtUtils::MakeMaker;. MakeMaker is the entry point to a large system of Perl modules. These modules know how to write makefiles, and various subclasses of them know how to write makefiles for your particular system. The WriteMakefile() call causes an appropriate makefile to be written. Arguments to WriteMakefile() control the contents of the makefile.

lib/Foo/Bar.pm

This is the actual Perl module. It is placed in a subdirectory tree that maps the module name, just as it will be when it is installed. If you add more modules to this build directory, you should place them on appropriate paths under lib/.

As created, Bar.pm contains some skeleton code and documentation; see Module Anatomy for a description of this.

N.B. Bar.pm is created with same copyright notice as README.

Building a Perl Module

To build a Perl module, you issue four commands from the module directory: 
perl Makefile.PL
make
make test
make install

perl Makefile.PL

The first step is to run Makefile.PL to create the makefile. Makefile.PL is a Perl program, but it has no #! line, so you have to invoke the interpreter explicitly.

make

This creates a directory named blib (for Build LIB) in the module directory, and installs the module in it. blib is a private test area for the module. Try doing 
.../development/Foo-Bar>ls -R blib/
Pretty amazing, no? If Foo::Bar linked to C code, you'd see those subdirectories populated with .ix files and .al files and .bs files and .so files and maybe a griffin or two. A copy of your .pm file is down there too: it's at 
.../development/Foo-Bar/blib/lib/Foo/Bar.pm

make test

This runs t/Foo-Bar.t, with special command line switches that cause Perl to find the copy of your module that is buried under blib/, and not any other copies that might be floating around. This is where you'll see syntax errors flagged, and functional errors if you added your own tests.

make install

This installs your module in /usr/local/lib/perl5/site_perl. On Unix systems, you typically need root access to do this. As soon as it's installed, it's available to everyone on your system.

make dist

If you want to share your module with users on other systems, do a 
.../development/Foo-Bar>make dist
This locates all the files listed in MANIFEST, tar's and compresses them, and writes the archive to (for example) 
Foo-Bar-1.00.tar.gz
in the development directory. This archive is a complete distribution of your module. You can give it to anyone, and they can build and install your module on their own system.

pm_to_blib

All good makefiles compare the modification times on the source and object files, and only rebuild things that are out of date. Tracking the modification times of all the files under blib/ would be complicated, so the makefile creates a zero-length file named pm_to_blib in the module directory and tracks its modification time instead.

If pm_to_blib gets out of sync with your .pm files, then the makefile will think blib/ is up to date and won't rebuild it. If this happens, just delete pm_to_blib and run make again.

Installing in non-standard places

On many systems, /usr/local/lib/perl5/site_perl is owned by root with 0755 permissions. This means that you must su to root in order to install a module there. If you lack root on your system, you can

Somewhere else

Here's how to install a Perl module in a non-standard place.

First, select a directory to hold your Perl modules and scripts. It doesn't strictly matter where the directory is located, but it must be somewhere that

Your home directory is probably a good choice; /usr/tmp is probably not. We'll write 
/home/uid
for your home directory.

Then go back to the development directory and type 

.../development/Foo-Bar>perl Makefile.PL INSTALL_BASE=/home/uid
.../development/Foo-Bar>make install
This will build the module and install it under your private directory, as follows: 
/home/uid/lib/perl5/     # modules
/home/uid/bin/           # scripts
/home/uid/man/           # man pages

PERL5LIB

Perl doesn't know to look for modules in your private directory; you have to tell it. There are several ways to do this. Perhaps the most appropriate in this context is to set the PERL5LIB environment variable. In Bourne shells, run 
export PERL5LIB=/home/uid/lib/perl5
and Perl will add your directory to the front of @INC.

Building Related Perl Modules

Sometimes you have several modules that work together. You could create a separate module directory for each one: 
.../development>h2xs -X -n Foo::Bar
.../development>h2xs -X -n Foo::Zot
One problem with this is that you have to keep hopping between directories to edit and build them. A bigger problem is that you can't test them together.

Suppose you make edits to both modules. Now you try to test Foo::Bar. Foo::Bar uses Foo::Zot. In particular, it uses the currently released version, which doesn't have the edits that you just made to Foo::Zot in the development area. So you can't test Foo::Bar until you've installed the new version of Foo::Zot, which you probably don't want to do right now, since you haven't tested it yet.

The solution is to create both modules into the same directory: 

.../development>h2xs -X -n Foo::Bar
.../development>cp Foo-Bar/lib/Foo/Bar.pm Foo-Bar/lib/Foo/Zot.pm
You don't even have to mess with Makefile.PL to do this; MakeMaker scans the build directory for .pm files and includes them all in the makefile.

Testing a Perl Module from the Build Directory 

.../development/Foo-Bar>make test
tests your module by running the test files in t/. h2xs creates t/Foo-Bar.t for you, but you can have as many .t files as you like. For example, if you are maintaining several .pm files in the same module directory, each one can have it's own .t file: 
Foo-Bar/t/Foo-Bar.t
Foo-Bar/t/Foo-Zot.t
Test files run in alphabetical order, so if you want to control the execution order, you can give them names like 
Foo/Bar/t/1_basic.t
Foo/Bar/t/2_advanced.t

Writing tests

make test uses Test::Harness to run the test files. Test files print their results on STDOUT. Test::Harness collects the output and reports the number of tests that pass and fail.

The TAP (Test Anything Protocol) documentation details the format of the output expected from test files. However, to get started, you only need to know a few things

The Test::Simple and Test::More modules provide additional facilities to aid in writing tests. I like to put 
my $N = 1;
sub Not { print "not " }
sub OK  { print "ok ", $N++, "\n" }
at the top of my .t files. Then I can write test code like 
$actual eq $expected or Not; OK;

Data files

If a module reads or writes files, you'd like to be able to test that functionality. t/ is a convenient place to put data files for test routines to use: 
.../development/Foo-Bar/t/Foo-Bar.t
.../development/Foo-Bar/t/fodder
make test will run Foo-Bar.t as usual; it will ignore fodder, because that file doesn't have a .t extension. Foo-Bar.t can then test Foo::Bar on the fodder file.

Beware that when make test executes a .t file, the current working directory is the module directory, not t/. So 

open FILE, "fodder"
won't work. Instead, you have to write 
open FILE, "t/fodder"

Verbose Output

If your module fails some of its tests, you'd probably like to know which ones. You can get more output from your test routines by running 
make TEST_VERBOSE=1 test
This will dump STDOUT from the test files to your screen. Some well chosen description text in the test output will make this more informative.

Selecting tests

By default, make test runs all the .t files in the t directory. If you are only interested in a single test file, you can run it with 
make TEST_FILES=t/Foo-Bar.t test
Switches to make test can be combined. To see the output from a single test file, do 
make TEST_FILES=t/Foo-Bar.t TEST_VERBOSE=1 test

Testing a module from a Perl program

Sometimes, you want to test a module by running a program that actually uses it. If the module is installed, this is easy enough, but what if the module isn't installed? Somehow, you have to get the program to find the .pm file under the module directory.

This used to require playing complicated games with @INC or use lib, but as of Perl 5.004, there's a better way: blib. blib is (what else?) a Perl module. If you are in the module directory, you can do 

.../development/Foo-Bar>perl -Mblib program.pl
to give program.pl access to Foo::Bar. If you are somewhere else, give the path to the module directory as an argument to blib 
somewhere/else>perl -Mblib=.../development/Foo-Bar program.pl

Documenting a Perl Module

Perl modules contain their own documentation, written in a simple markup language called Plain Old Documentation (POD). There are a variety of filters for translating POD files to other formats: pod2text, pod2man, pod2html, pod2latex, pod2usage.

Programmers are strongly encouraged to write PODs for all their Perl programs and modules. See module PODs for details.

Man Pages

Man pages for Perl modules are stored under man/man3/.

The build procedure for a Perl module automatically creates and installs a man page for the module. make invokes pod2man to generate Foo::Bar.3 from Bar.pm, and make install copies Foo::Bar.3 to the man3 directory.

Notes

unique
Strictly speaking, module names need only be unique within each installation of the Perl interpreter. However, module writers are strongly encouraged to choose globally unique names for their modules.
single
XS and autosplit modules comprise multiple files.
INC
Arguably, it would be better named @LIB, but...um...it isn't.
current working directory
This is less useful than it seems, because you aren't usually in the directory where the module is stored.
users lined up at your door
I speak from personal experience here.
XS
XS is a (phonetic?) acronym for eXternal Subroutine. "External" means external to Perl, i.e. written in C. For more information on XS, see XS Mechanics
floating around
say, in /usr/local/lib/perl5/site_perl/
out of sync
say, because you touched it, or because of NFS latency
mess with Makefile.PL
MakeMaker is large and complex, and the makefiles that it generates are inscrutable. Caveat Scriptor.

Translations

Russian translation courtesy of Akhmad Karimov
Steven W. McDougall / resume / swmcd@theworld.com / 2007 March 02