Requesting compressed content from a web server with LWP::UserAgent

This article explains how to ask LWP::UserAgent to request compressed content to save bandwidth. The following Perl script demonstrates a download with and without compression enabled. Compression is requested using the Accept-Encoding header. The line

$ua->default_header('Accept-Encoding' => 'gzip');

turns on this header for LWP::UserAgent. The server replies using the Content-Encoding header, which is got from LWP::UserAgent using the following line.

my $content_encoding = $response->header ('Content-Encoding');

#!/usr/local/bin/perl
use warnings;
use strict;
use LWP::UserAgent;
use Gzip::Faster;

my $ua = LWP::UserAgent->new ();
my $url = 'http://www.lemoda.net/games/figlet/figlet.cgi?text=frog';
get ($ua, $url);
$ua->default_header('Accept-Encoding' => 'gzip');
get ($ua, $url);
exit;

sub get
{
    my ($ua, $url) = @_;
    my $response = $ua->get ($url);
    if ($response->is_success ()) {
        my $content_encoding = $response->header ('Content-Encoding');
        my $text = $response->content;
        if ($content_encoding) {
            print "Content encoding is $content_encoding.\n";
            if ($content_encoding eq 'gzip') {
                my $uncompressed = gunzip ($text);
                printf "Uncompressed from %d bytes to %d bytes.\n",
                    length $text, length $uncompressed;
                $text = $uncompressed;
            }
        }
        else {
            print "Content encoding is not set.\n";
        }
        print $text;
    }
    else {
        print STDERR "get '$url' failed: ", $response->status_line, "\n";
    }
}

(download)

The output should look like

Content encoding is not set.
  __                 
 / _|_ __ ___   __ _ 
| |_| '__/ _ \ / _` |
|  _| | | (_) | (_| |
|_| |_|  \___/ \__, |
               |___/ 
Content encoding is gzip.
Uncompressed from 85 bytes to 132 bytes.
  __                 
 / _|_ __ ___   __ _ 
| |_| '__/ _ \ / _` |
|  _| | | (_) | (_| |
|_| |_|  \___/ \__, |
               |___/ 

Alternatively, you can use the built-in facilities of libwww-perl. You need to explicity create a HTTP::Request object, like this:

#!/usr/local/bin/perl
use warnings;
use strict;
use LWP::UserAgent;
use HTTP::Request;
my $url = 'http://www.lemoda.net/games/figlet/figlet.cgi?text=frog';
my $request = HTTP::Request->new (GET => $url);
my $ua = LWP::UserAgent->new ();
get ($ua, $request);
$request->accept_decodable;
get ($ua, $request);

sub get
{
    my ($ua, $request) = @_;
    my $response = $ua->request ($request);
    if ($response->is_success ()) {
        my $text = $response->content;
        my $uncompressed = $response->decoded_content;
        if ($text ne $uncompressed) {
            print "Response was compressed.\n";
        }
        else {
            print "Response was not compressed.\n";
        }
        print "$uncompressed\n";
    }
    else {
        print STDERR "get '$url' failed: ", $response->status_line, "\n";
    }
}

This should print out the following:

Response was not compressed.
  __                 
 / _|_ __ ___   __ _ 
| |_| '__/ _ \ / _` |
|  _| | | (_) | (_| |
|_| |_|  \___/ \__, |
               |___/ 

Response was compressed.
  __                 
 / _|_ __ ___   __ _ 
| |_| '__/ _ \ / _` |
|  _| | | (_) | (_| |
|_| |_|  \___/ \__, |
               |___/ 

However, this has the side effect of also decoding any UTF-8 strings in the output into Perl's Unicode encoding, which may or may not be what you want to do.