To open a file in read mode, use the < sign for the mode. Here’s what it looks like:

my $filename = 'chapter_9/targets.txt';
open my $spies_to_espy, '<', $filename
    or die "Cannot open '$filename' for writing: $!";

That is a lot of new stuff at once, so we will break it down carefully.

The my $spies_to_espy variable contains the filehandle that you will use to access the contents of $filename. Like variables, a filehandle with a descriptive name leads to clearer code. You will find that filehandle is commonly abbreviated at $fh.

The < tells Perl you’re going to open the file for reading.

If the attempt to open the file fails the open() builtin returns false and sets the special $! variable. $! contains a human-readable description of the error. You can print $! to provide an error message. If the above file does not exist, that might print:

Cannot open 'chapter_9/targets.txt' for writing: No such file or
directory at my_program.pl line 17.

When using open() and other related functions, always include the or die section at the end. Otherwise, Perl may ignore the error and silently Do The Wrong Thing and that would be disappointing. To automate remembering you can install the handy autodie module from the CPAN and it will take care of this for you:

use autodie;
my $filename = 'chapter_9/targets.txt';
open my $filehandle, '<', $filename;

If open() fails, you’ll get a virtually identical error message to the one above.

The autodie module was included with Perl as of version 5.10.1, so if you have that version of Perl or newer, you won’t need to install it separately.

my $filename = "chapter_9\targets.txt";

my $filename = "chapter_9\\targets.txt";

my $file = "path\\to\\some\\$file";

my $file = "path/to/some/$file";

my $file = "C:/path/to/some/$file";

Now that we have opened the file, let’s read from it and print the name, case number and description of each record. Example 9.1, “Reading a parsing a file” shows the code to do this.

use strict;
use warnings;
use diagnostics;
my $filename = 'chapter_9/targets.txt';
open my $spies_to_espy, '<', $filename
    or die "Cannot open '$filename' for writing: $!";
while ( my $line = <$spies_to_espy> ) {
    next if $line =~ /^\s*#/; # skip comments!
    chomp($line);
    my ( $name, $case_number, $description )
      = split /\|/, $line;
    print "$name ($case_number): $description\n";
}
close $spies_to_espy or die "Could not close '$filename': $!";

And that will print out:

James (007): Spy
Number 6 (6): Ex-spy
Agent 99 (99): Spy with unknown name
Napoleon Solo (11): Uncle spy
Unknown (666): Maybe a spy

We’ll take this line-by-line so you can clearly see what is going on here.

while ( my $line = <$spies_to_espy> ) {

The angle brackets around the filehandle turn it into an iterator. If you assign it in list context (such as assigning it to an array), it will read in every record in the file, as separated by the value in the $/ variable. If you assign it to a scalar, as shown above, it acts like an iterator, returning one line at a time, or undef when there is no more input. As you will recall from Chapter 5, Control flow, we usually use while loops with iterators.

while ( my $line = <$fh> ) { ... }
# becomes
while ( defined ( my $line = <$fh> ) ) { ... }

The $/ variable defaults to whatever the newline character is for your operating system. For Windows, this is the carriage return plus line feed (\r\n). For Unix-like systems such as Linux, Mac OS X, AIX and so on, it’s just the line feed character (\n) and for versions of Mac OS prior to OS X, it’s just the carriage return (\r). Other operating systems may use different characters, but Perl takes care of this for you.

my $file_contents = slurp('chapter_9/targets.txt');
print $file_contents;
sub slurp {
    my $file = shift;
    open my $fh, '<', $file
      or die "Cannot open '$file' for reading: $!";
    local $/;
    my $contents = <$fh>;
    return $contents;
}

That’s written for clarity. However, you’ll often see it written like this:

sub slurp {
    my $file = shift;
    open my $fh, '<', $file
      or die "Cannot open '$file' for reading: $!";
    return do { local $/; <$fh> };
}

The next line of code should be clear. We skip comments in the file by preceding them with a # symbol. The \s* allows you to have zero or more spaces in front of the # symbol.

next if $line =~ /^\s*#/; # skip comments!

Then we have the chomp():

chomp($line);

As you may recall from Chapter 4, Working With data, chomp() removes anything matching $/ from the end of the variable. In this case, we don’t need to do this because we’re adding it back in when we print the data, It is a very good habit to get into. You often store data in variables and probably do not want the line separator.

Then we split the line on the pipe character, |. Because split() expects a regular expression as its first argument and the | is used for alternation, we need to escape it to match a literal pipe character.

my ( $name, $case_number, $description )
      = split /\|/, $line;

And finally we print our results:

print "$name ($case_number): $description\n";

The final line closes our filehandle:

close $fh or die "Could not close '$filename': $!";

Note that if the filehandle falls out of scope, Perl will close the filehandle for you. You’ll see many programs take advantage of this feature and not close their filehandles.

Note that the <> operator assigns to $_ by default, so you can omit the my $line = if you prefer:

while (<$fh>) {
    next if /^\s*#/; # skip comments!
    chomp;
    my ( $name, $case_number, $description ) = split /\|/, $_;
    print "$name ($case_number): $description\n";
}

For versions of Perl prior to version 5.6 (released over a decade ago!), you will often see this syntax:

open FH, $filename
  or die "Cannot open '$filename" for reading: $!";

Or:

open FH, "< $filename"
  or die "Cannot open '$filename" for reading: $!";

This combines a few practices that are today considered very bad. The FH looks like a bareword and should not be allowed with use strict, but in this instance, it’s considered to be a typeglob. You use it like a normal filehandle:

while ( my $line = <FH> ) { ... }

This is considered bad practice because typeglobs are global and there can be some very strange bugs associated with other portions of your program messing with global variables. Imagine trying to debug what’s going wrong with this:

open FH or die $!;

That’s perfectly legal and it might just open a file in read mode, but we won’t cover this monstrosity here (again, see perlopentut for the gory bits).

Note that we’re also using the two argument form of open() in this bad example:

open FH, $filename;
# and
open FH, "< $filename";

For the first, we’ve simply omitted the < mode. If that’s left off, Perl assumes read mode. For the second, it’s included in the string, along with the filename. That does the same thing. It has to do with making this seem a bit more familiar to Unix programmers, but suffice it to say that it’s strongly discouraged today. Some operating systems allow filenames to start with characters that might be interpreted by Perl as changing the mode of the file open. Thus, in the good ol’ days, a simple open FH, $filename may have very unexpected behavior.

Don’t do that. Stick with the three argument open.

Writing files has a similar syntax, but we use > to open the file in “write mode”. If you wish to append to a file, use >>. So to add Maxwell Smart as a new “target” in targets.txt, you could write the following:

open my $fh, '>>', $filename
  or die "Cannot open '$filename' for appending: $!";
print $fh "Maxwell Smart|86|Definitely a spy\n";

And now the file should contain:

James|007|Spy
Number 6|6|Ex-spy
# This guy is only rumored to exist. Not everyone believes it.
Unknown|666|Maybe a spy
Maxwell Smart|86|Definitely a spy

Nothing unusual about this, except for the print line:

print $fh "Maxwell Smart|99|Definitely a spy\n";

Note that there’s no comma after the $fh. That’s what let’s Perl know that $fh is a file handle it’s printing to instead of something to print. So if you see something like this when you weren’t expecting any output:

GLOB(0xbfe220)Maxwell Smart|99|Definitely a spy

You probably put a comma after the filehandle, telling Perl that it’s something to print instead of a filehandle to print to.

If you want, you can rewrite the file by reading it and then writing to it. Let’s sort the lines of the file and strip the comments from it. Here’s one way of doing that.

my $filename = 'chapter_9/targets.txt';
open my $fh, '<', $filename
  or die "Cannot open '$filename' for reading: $!";
# each element in @lines gets one line from the file
# remember grep from Chapter 4?
my @lines = sort grep { !/^\s*#/ } <$fh>;
close $fh or die "Cannot close '$filename': $!";
open my $fh, '>', $filename
  or die "Cannot open '$filename' for writing $!";
print $fh @lines;
close $fh or die "Cannot close '$filename': $!";

Once again, this code is building on everything you’ve learned so far. There’s nothing to magical here.

There is another way of rewriting a file. We need four things: seek(), tell(), truncate() and read-write mode.

To open a file in read-write mode, prepend the mode with a +. In this case, we’ll use +< mode. There is a corresponding +> mode, but you should probably never use it because it will delete the contents of your file first. That’s probably not very helpful. Here’s our new program.

my $filename = 'chapter_9/targets.txt';
open my $fh, '+<', $filename
  or die "Cannot open '$filename' in read-write mode: $!";
my @lines = sort grep { !/^\s*#/ } <$fh>;
seek $fh, 0, 0
  or die "Cannot seek '$filame', 0, 0: $!";
print $fh @lines;
truncate $fh, tell($fh)
  or die "Cannot truncate '$filename': $!";
close $fh or die "Cannot close $filename: $!";

The seek() function has the following syntax:

seek FILEHANDLE, OFFSET, STARTINGAT

The values for STARTINGAT are

The tell() function returns the position of the filehandle, in bytes. The truncate() builtin tells Perl to truncate the file at the given position.

This may seem a bit confusing, but it’s what Perl needs to know to handle this. And again, don’t forget that you can use autodie to make this simpler

use autodie;
my $filename = 'chapter_9/targets.txt';
open my $fh, '+<', $filename;
my @lines = sort grep { !/^\s*#/ } <$fh>;
seek     $fh, 0, 0;
print    $fh  @lines;
truncate $fh, tell($fh);
close    $fh;

Though your author usually uses autodie, we’re avoiding it in examples to constantly remind you to check the success or failure of your system calls. As usual, see perldoc -f for the various functions to learn more about them.

Decoding your data means “decode your data to Perl’s internal format”. What is Perl’s internal format? It doesn’t matter. If Perl ever needs to change that internal format, you should not be relying on knowing the details. Suffice it to say that Perl will generally treat your text data as binary data instead of characters until you decode it and write it out somewhere. This is the hard part. You have to find out what the encoding of your source data is! So if your data is in 7bit-jis (a Japanese pre-Unicode encoding), you could use the Encode::decode() function to transform it into Perl’s internal format:

use Encode qw(encode decode);
my $string = decode('7bit-jis', $byte_string);

And now Perl will happily handle this for you, including reporting its length correctly.

However, it’s better to not have to decode strings on a string-by-string basis. It’s better to decode them at the source, if possible (thus making it harder to forget). You can use Perl’s IO layers to handle that. One way is to specify the layer with binmode():

open my $fh, '<', $some_file or die $!;
binmode $fh, ':7bit-jis';

Or better still, specify it with the mode because it’s harder to miss:

open my $fh, '<:7bit-jis', $some_file or die $!;

If you don’t know the encoding of your source data, ask the person who sent you the data. If that fails, Encode (first shipped with Perl 5.7.3) includes the Encode::Guess module. It’s not a bad module, but it’s a “guess” at the encoding. Read the documentation carefully and be aware that it will guess wrong from time to time.

Now that you’ve decoded your data and done fun things with it, you need to encode it back to its original format before you send it along. Not surprisingly, the encode() function from Encode does this for you.

use Encode qw(encode decode);
my $encoded = encode('7bit-jis', $string);

Or again, use the IO layers:

open my $fh, '>:7bit-jis', $some_file or die $!;

Then, when you write the data out to the console, a file or some other data sink, it will be encoded correctly.

So decode your input and encode your output. Not too bad, right? Well, that’s until you try it. First, let’s look at this code snippet.

my $string = '日本国';
my $length = length($string);
print "$string has $length characters\n";

And that prints out (assuming you have the correct font installed):

日本国 has 9 characters

Of course, that’s not true. It has 9 octets, but it clearly has 3 characters. So the first thing that many people do is this:

use utf8;
my $string = '日本国';
my $length = length($string);
print "$string has $length characters\n";

Many people assume that use utf8 means “magically make everything UTF-8”, but that’s not correct. We get the following output:

Wide character in print at /var/tmp/eval_Yrhm.pl line 4.
日本国 has 3 characters

Note the strange Wide character in print warning, but we now have the correct length. The use utf8 pragma only tells Perl that our source code is utf8. It doesn’t tell Perl that your output is UTF-8, so Perl is expecting a binary output to the STDOUT filehandle but you’ve sent UTF-8, so let’s fix that.

use utf8;
my $string = '日本国';
my $length = length($string);
binmode STDOUT, ':encoding(UTF-8)';
print "$string has $length characters\n";

And that will give us the correct output with no warnings (note that the Wide Character in Print warning will occur even if you don’t use warnings).

Alternatively, if you don’t want to force STDOUT to be UTF-8, you could just encode the string from Perl’s internal format to UTF-8 and this will also make the warning go away.

use utf8;
use Encode qw(encode decode);
my $string = '日本国';
my $length = length($string);
$string    = encode('UTF-8', $string);
print "$string has $length characters\n";

But we’re still not quite where we want to be in understanding this. The use utf8 pragma tells Perl that your source code is UTF-8, bit it doesn’t tell Perl that your input is UTF-8. Try this:

use utf8;
use Encode qw(encode decode);
my $string = shift @ARGV;
my $length = length($string);
$string    = encode('UTF-8', $string);
print "$string has $length characters\n";

If you save that as length.pl and run that with perl length.pl 日本国, you will get output similar to:

æ¥æ¬å½ has 9 characters

You won’t even get a warning. Why? Because you haven’t decoded the data and Perl assumes it’s Latin-1 data (ISO-8859-1) that it already knows how to deal with. When you explicitly decode the data, everything works as expected:

use utf8;
use Encode qw(encode decode);
my $string = decode('UTF-8', shift);
my $length = length($string);
$string    = encode('UTF-8', $string);
print "$string has $length characters\n";

If you are unsure of what encodings your system provides, the following one-liner will print all of them for you:

perl -MEncode -e 'print join "\n" => Encode->encodings(":all")'

binmode STDOUT, ':utf8';

open my $fh, '<:utf8', $filename;

Case folding is converting all of the characters in a string to upper or lower case. This is useful when you want to make case-insensitive comparisons. It’s also often a dangerous thing to do with Unicode. Consider the following program:

use utf8;
binmode(STDOUT, ":encoding(UTF-8)");
print uc("σ"), "\n";   # Greek small letter sigma
print uc("ς"), "\n";   # Greek small final letter sigma

That will print out the same letter twice, an upper-case sigma character:

Σ
Σ

The σ and ς characters are the same lower-case sigma character, but the latter is used at the end of the word. When you call uc() on them, they both resolve to an upper-case sigma, Σ. This leads to this problem:

use utf8;
binmode(STDOUT, ":encoding(UTF-8)");
print lc(uc("σ")), "\n";   # Greek small letter sigma
print lc(uc("ς")), "\n";   # Greek small final letter sigma

That will print σ twice, meaning that case-folding is not round-trip safe in Unicode.

In fact, in earlier versions of Perl, in some cases, characters in the range 128 to 255 would often have strange behavior when you tried to use lc, uc, ucfirst, and so on. When used as characters, they would sometimes be considered Unicode code points and when used as bytes, they could be considered “unassigned characters” and not match \w in regular expressions. The solution is simple:

use feature 'unicode_strings';

Unfortunately, that feature was not added until Perl 5.11.3 (a development release). So today it’s argued that you should really use Perl 5.12 or better (preferably 5.14) if you really want to be “Unicode safe”.

You need to convert between UTF-16 and ISO-8859-1 (Latin-1). To do this, you must convert from one encoding to Perl’s internal format and then convert to the desired format:

my $string = decode('UTF-16', $utf16_data);
my $latin1 = encode('iso-8859-1', $string);

However, ISO-8859-1 is a subset of UTF-16, so you may lose data.

You’ll see this warning a lot when you’re working with character encodings and you’re not being careful. When this happens, it’s because you haven’t specified your encoding layer. Perl then assumes your data is ISO-8859-1 (for backwards-compatibility) and tries to output UTF-8. Any data that doesn’t fit in the ISO-8859-1 range emits this warning. That’s why you got this warning with this code snippet we used earlier:

use utf8;
my $string = '日本国';
my $length = length($string);
print "$string has $length characters\n";

The input data may be read from files, the command line, sockets, and other data sources. The output data may be written to STDOUT, files, or other data sinks. To tell Perl that all input and output data is UTF-8, you can set the PERL_UNICODE environment variable to AS. The A and S letter combination is described in the -C section of perldoc perlrun.

Unfortunately, it’s not as simple as setting the environment variable in your code. You must set this before you run your program. On a Linux style system, you can do this:

PERL_UNICODE=AS perl program.pl

Or you can export the variable and it will be set for all programs:

export PERL_UNICODE=AS

On Windows, the syntax is:

set PERL_UNICODE=AS

This can be a hassle to do every time and it may very well be the wrong thing to do if you have non-UTF-8 data.

Sometimes you’ll see this in code:

use Encode 'is_utf8';
if ( is_utf8($string) ) {
    # wrong!
}

Or the identical:

if ( utf8::is_utf8($string) ) {
    # wrong!
}

This does not work as you think it does. The is_utf8() function is used internally to determine if Perl should treat a string as Latin-1 or UTF-8. However, just because the utf8 flag is set does not mean that the string is actually UTF-8. Like the Encode::Guess module, it’s just a guess (for you) and you explicitly set your encoding layers as described earlier.

If you want a shortcut for assuming that @ARGV, your filehandles and your source code are all UTF-8, you can install the utf8::all module from the CPAN.

use utf8::all;

You may recall this program from earlier:

use utf8;
use Encode qw(encode decode);
my $string = decode('UTF-8', shift);
my $length = length($string);
$string    = encode('UTF-8', $string);
print "$string has $length characters\n";

With the utf8::all pragma, that becomes:

use utf8::all;
my $string = shift @ARGV;
my $length = length($string);
print "$string has $length characters\n";

In other words, it makes it easier to write programs with UTF-8 data. It’s not perfect, but it’s a good start.

By now you already know how to open your STDOUT to handle printing Unicode, but what about typing those funny characters? Well, you don’t have to. One way of avoiding this is the charnames pragma:

use utf8::all;
use charnames ':short';
# note that double-quoted strings are required
print "\N{greek:Sigma} is an upper-case sigma.\n";

And that prints (with no warning due to utf8::all):

Σ is an upper-case sigma.

The \N{} construct with charnames is resolved at compile-time, so you cannot use variables there.

You can also use the Unicode fullnames:

use utf8::all;
use charnames ':full';
print "\N{GREEK SMALL LETTER ETA WITH DASIA AND PERISPOMENI}\n";

Which prints ἧ.

If you know the code point but not the name, you can use \N{U+codepoint}. Again, remember this is done at compile time. Thus, the code point for the smiley face character is U+263A, so you can print it with this:

use utf8::all;
print "\N{U+263A}\n";

Or you can just fall back to the chr() function:

print chr(0x263a);

See http://unicode.org/charts/ for a list of the appropriate names you may wish to print.

The character ἧ is a Greek letter, but is it upper or lower case? You can try Unicode character properties to find out:

use utf8::all;
my $character ='ἧ';
if ( $character =~ /\p{Lowercase}/ ) {
    print "$character is lower case\n";
}
if ( $character =~ /\p{Uppercase}/ ) {
    print "$character is upper case\n";
}

That correctly prints ἧ is lower case.

Unicode properties are properties about characters that describe something about it. They might describe the case of the letter, the script used, whether or not it’s a math symbol or punctuation and so on. Unicode is so all-encompasing — and it must be since it is trying to handle all writing systems — that you will find many strange things in Unicode land. Here’s one of them:

use utf8::all;
# latin capital letter d with small letter z
my $character = "\N{U+01F2}";
if ( $character =~ /\p{Lowercase}/ ) {
    print "$character is lower case\n";
}
if ( $character =~ /\p{Uppercase}/ ) {
    print "$character is upper case\n";
}
if ( $character =~ /\p{Titlecase_Letter}/ ) {
    print "$character is title case\n";
}

And that prints:

Dz is title case

This is because the Latin capital letter d with small letter z is considered a Titlecase character and is not upper or lower case. Fun, eh?

You can spend a long time understanding Unicode and this section of the book is far too short, but here a couple of good starting points for understanding Unicode and some of the associated issues.

First, read Joel Spolsky’s famous “The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)” article at http://www.joelonsoftware.com/articles/Unicode.html

Second, read this:

http://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default

In that link, Tom Christiansen explains, in-depth, many of the traps to be aware of. It’s mind-bending, but it begins to give you an idea of what you’re up against.

Also, http://en.wikipedia.org/wiki/Free_software_Unicode_typefaces has a list of Free Unicode fonts you can install if you’re tired of seeing broken characters when you try to print Unicode.