This is the official version of Apache Pig. It comes packaged with all of the jars needed to run Pig. It can be downloaded by going to Pig's release page.

Pig does not need to be installed on your Hadoop cluster. It runs from the machine you launch Hadoop jobs from. Though you can run Pig from your laptop or desktop, in practice most cluster owners set up one or more machines that have access to their Hadoop cluster but are not part of the cluster (that is, they are not data nodes or task nodes). This makes it easier for administrators to update Pig and associated tools, and to secure access to the clusters. These machines are called gateway machines or edge machines. In this book I will use the term gateway machine.

It is on these gateway machines that you will need to install Pig. If your Hadoop cluster is accessible from your desktop or laptop then you can install Pig there as well. Also, you can install Pig on your local machine if you plan to use Pig in local mode.

The core of Pig is written in Java, and is thus portable across operating systems. The shell script that starts Pig is a bash script, so it requires a Unix environment. Hadoop, which Pig depends on even in local mode, also requires a Unix environment for its file system operations. In practice, most Hadoop clusters run a flavor Linux. Many Pig developers develop and test Pig on MacOS X.

Pig requires Java 1.6. And Pig versions 0.5 through 0.9 require Hadoop 0.20. For future versions, check the download page for information on what version(s) of Hadoop they require. The correct version of Hadoop is included with the Pig download. If you plan to use it in local mode or install it on a gateway machine where Hadoop is not currently installed, there is no need to download Hadoop separately.

Once you have downloaded Pig, you can place it anywhere you like on your machine. Pig does not depend on being in a certain location. To install it, place the tarball in the directory of your choosing and type:

tar xzf filename

where filename is the tar file you downloaded.

The only other setup in preparation for running Pig is making sure that the environment variable JAVA_HOME is set to the directory that contains your Java distribution. Pig will fail immediately if this value is not in the environment. You can set this in your shell, specify it on the command line when you invoke Pig, or set it explicitly in your copy of the Pig script pig, located in the bin directory that you just unpacked. You can find the appropriate value for JAVA_HOME by doing which java and stripping the bin/java from the end of the result.

In addition to the official Apache version, there are companies that repackage and distribute Hadoop and associated tools. The most popular of these currently is Cloudera. They produce RPMs for Red Hat based systems and packages for use with APT on Debian systems. They also provide tarballs for other systems that cannot use one of these package managers.

The upside of using a distribution like Cloudera's is that all of the tools are packaged and tested together. Also, if you need professional support, it is available. The downside is that you are constrained to move at the speed of your distribution provider. There is a delay between an Apache release of Pig and its availability in various distributions.

For complete instructions on downloading and installing Hadoop and Pig from Cloudera, see Cloudera's download site. Note that you have to download Pig separately. It is not part of the Hadoop package.

In addition to the official release available from Pig's Apache site, it is possible to download Pig from Apache's Maven repository. This site includes jar files for Pig, for the source code, for the javadocs, and the pom file that defines Pig's dependencies. Development tools that are Maven aware can use this to pull down Pig's source and javadoc. If you use maven or ant in your build process you can also pull the Pig jar automatically from this repository.

When you download Pig from Apache you also get the Pig source code. This enables you to debug your version of Pig or just to peruse the code to see how it works. But if you want to live on the edge and try out a feature or a bug fix before it is available in a release, you can download the source from Apache's Subversion repository. You can also apply patches that have been uploaded to Pig's issue tracking system but not yet checked into the code repository. Information on checking out Pig using svn or cloning the repository via git is available on Pig's version control page.

Running Pig locally on your machine is referred to in Pig parlance as local mode. Local mode is useful for prototyping and debugging your Pig Latin scripts. Some people also use it for small data when they want to apply the same processing to large data, so that their data pipeline is consistent across data of different sizes, but they do not want to waste cluster resources on small files and small jobs.

In versions 0.6 and before, Pig executed scripts in local mode itself. Starting with version 0.7, it uses a Hadoop class LocalJobRunner that reads from the local file system and executes MapReduce jobs locally. This has the nice property that Pig jobs run locally in the same way as they will on your cluster. And they run all in one process. This makes debugging much easier. The downside is that it is slow. Setting up a local instance of Hadoop has approximately a twenty second overhead. So even tiny jobs take at least that long.^[1]

--average_dividend.pig
-- load data from NYSE_dividends, declaring the schema to have 4 fields
dividends = load 'NYSE_dividends' as (exchange, symbol, date, dividend);
-- group rows together by stock ticker symbol
grouped   = group dividends by symbol;
-- calculate the average dividend per symbol
avg       = foreach grouped generate group, AVG(dividends.dividend);
-- store the results to average_dividend
store avg into 'average_dividend';
				

NYSE    CPO 2009-12-30  0.14
NYSE    CPO 2009-09-28  0.14
NYSE    CPO 2009-06-26  0.14
NYSE    CPO 2009-03-27  0.14
NYSE    CPO 2009-01-06  0.14

CA      0.04
CB      0.35
CE      0.04
CF      0.1
CI      0.04

Most of the time you will be running Pig on your Hadoop cluster. As was covered above, the section called “Downloading and Installing Pig”, Pig runs locally on your machine or your gateway machine. All of the parsing, checking, and planning is done locally. It then executes MapReduce jobs in your cluster.

The only thing Pig needs to know to run on your cluster is where your cluster's NameNode and JobTracker are located. The NameNode is the manager of HDFS and the JobTracker coordinates MapReduce jobs. In Hadoop 0.18 and before, these locations are found in your hadoop-site.xml file. In Hadoop 0.20 and later they are in separate files hdfs-site.xml and mapred-site.xml.

If you are already running Hadoop jobs from your gateway machine via MapReduce or another tool then you most likely have these files present. If not, the best course is to copy these files from nodes in your cluster to a location on your gateway machine. This guarantees that you get the proper addresses plus any site specific settings.

If, for whatever reason, it is not possible to copy the appropriate files from your cluster, you can create a hadoop-site.xml file yourself. It will look like:

<configuration>
<property>                                                                     
  <name>fs.default.name</name>                                                 
  <value>namenode_hostname:port</value>                         
</property>

<property>                                                                     
  <name>mapred.job.tracker</name>
  <value>jobtrack_hostname:port</value>                        
</property>
</configuration>

You will need to find the names and ports for your NameNode and JobTracker from your cluster administrator.

Once you have located, copied, or created these files, you will need to tell Pig the directory they are in by setting the PIG_CLASSPATH environment variable to that directory. Note that this must point to the directory that the XML file is in, not the file itself. Pig will read all XML and properties files in that directory.

Cloud computing^[2] along with the software as a service (SaaS) model has taken off in recent years. This has been fortuitous for hardware intensive applications like Hadoop. Setting up and maintaining a Hadoop cluster is an expensive proposition in terms of hardware acquisition, facility costs, and maintenance and administration. Many users find that it is cheaper for them to rent the hardware they need instead.

Whether you or your organization decides to use Hadoop and Pig in the cloud or on owned and operated machines, the instructions for running Pig on your cluster are the same, the section called “Running Pig on Your Hadoop Cluster”.

Amazon's Elastic MapReduce (EMR) is a cloud offering however which is different. Rather than allowing customers to rent machines for any type of process (like Amazon's Elastic Cloud Computing (EC2) service and other cloud services), EMR allows users to rent virtual Hadoop clusters. These clusters read data from and write data to Amazon's Simple Storage Service (S3). This means users do not even need to set up their own Hadoop cluster as they would if they used EC2 or a similar service.

EMR users can access their rented Hadoop cluster via their browser, ssh, or a web services API. EMR is at http://aws.amazon.com/elasticmapreduce/. I suggest beginning with the nice tutorial which will introduce you to the service.

Pig has a number of command line options that you can use with it. You can see the full list by doing pig -h. Most of these options will be discussed in the sections that cover the features these options control. In this section I will discuss the remaining miscellaneous options.

Pig also has a number of Java properties it uses. The entire list can be printed out by doing a pig -h properties. Specific properties will be covered in sections that cover the features they control.

Hadoop also has a number of Java properties is uses to determine its behavior. For example, you can pass options to the JVM that runs your map and reduce tasks by setting mapred.child.java.opts. In Pig version 0.8 and later these can be passed to Pig and Pig will pass them on to Hadoop when it invokes Hadoop. In earlier versions the properties had to be in the hadoop-site.xml so that the Hadoop client itself would pick them up.

Properties can be passed to Pig on the command line using -D in the same format as any Java property. For example, bin/pig -Dexectype=local. When placed on the command line, these property definitions must come before any Pig specific command line options (such as -x local). They can also be specified in the conf/pig.properties file that is part of your Pig distribution. Finally, you can specify a separate properties file by using -P. If properties are specified on both the command line and in a properties file, the command line specification takes precedence.

Pig uses return codes to communicate success or failure. Table 2.1, “Pig Return Codes” describes these return codes.