Calling the static method Pig.compile causes Pig to do an initial compilation of the code. Since we have not bound the variables yet, this check cannot completely verify the script. Type checking and other semantic checking is not done at this phase, only the syntax is checked. compile returns a Pig object that can be bound to a set of variables.

# pagerank.py
P = Pig.compile("""
previous_pagerank = load '$docs_in' as (url:chararray, pagerank:float,
                      links:{link:(url:chararray)});
outbound_pagerank = foreach previous_pagerank generate
                      pagerank / COUNT(links) as pagerank,
                      flatten(links) as to_url;
cogrpd            = cogroup outbound_pagerank by to_url,
                      previous_pagerank by url;
new_pagerank      = foreach cogrpd generate group as url,
                      (1 - $d) + $d * SUM (outbound_pagerank.pagerank)
                      as pagerank,
                      flatten(previous_pagerank.links) as links,
                      flatten(previous_pagerank.pagerank) AS previous_pagerank;
store new_pagerank into '$docs_out';
nonulls           = filter new_pagerank by previous_pagerank is not null and
                        pagerank is not null;
pagerank_diff     = foreach nonulls generate ABS (previous_pagerank - pagerank);
grpall            = group pagerank_diff all;
max_diff          = foreach grpall generate MAX (pagerank_diff);
store max_diff into '$max_diff';
""")

The only thing in this Pig Latin script that we have not seen before is the four parameters, marked in the script as $d, $docs_in, $docs_out, and $max_diff. The syntax for these parameters is the same as for parameter substitution. However, Pig expects these to be supplied by the control flow script when bind is called.

There are three other compilation methods in addition to the one shown in this example. compile(String name, String script) takes a name in addition to the Pig Latin to be compiled. This name can be used in other Pig Latin code blocks to import this block.

P1 = Pig.compile("initial", """
A = load 'input';
...
""")
    P2 = Pig.compile("""
import initial;
B = load 'more_input';
...
""")

The other two compilation methods are called compileFromFile. These take the same arguments as compile, but they expect the script argument to refer to a file containing the script, rather than the script itself.

Once your script has successfully been compiled, the next step is to bind variables in the control flow to variables in Pig Latin. In our example script this is done by providing a map to the bind call. The keys are the name of the variables in Pig Latin. The values in this example are literal string values that are updated as the script progresses. They could also be references to Python variables.

# pagerank.py
params = { 'd': '0.5', 'docs_in': 'data/webcrawl' }

for i in range(10):
    out = "out/pagerank_data_" + str(i + 1)
    max_diff = "out/max_diff_" + str(i + 1)
    params["docs_out"] = out
    params["max_diff"] = max_diff
    Pig.fs("rmr " + out)
    Pig.fs("rmr " + max_diff)
    bound = P.bind(params)
    stats = bound.runSingle()
    if not stats.isSuccessful():
        raise 'failed'
    mdv = float(str(stats.result("max_diff").iterator().next().get(0)))
    print "max_diff_value = " + str(mdv)
    if mdv < 0.01:
        print "done at iteration " + str(i)
        break
    params["docs_in"] = out

For the initial run, the Pig Latin$d will take on the value of 0.5, $docs_in the filename webcrawl, $docs_out out/pagerank_data_1, and $max_diff out/max_diff_1.

bind returns a BoundScript object. This object can be run, explained, described, or illustrated. As is shown in this script, a single Pig object can be bound multiple times. A compile is only necessary on the first pass, with different values being bound to it each time.

In our example bind is given a mapping of the variables to bind. If all of your Python variables and Pig Latin variables have the same name, then you can call bind with no arguments. This will cause bind to look in the Python context for variables of the same name as the parameters in Pig and use them. If it cannot find appropriate variables, then it will throw an error. We could change our example script to look like this:

# pagerankbindnoarg.py
d = 0.5
docs_in = 'data/webcrawl'

for i in range(10):
    docs_out = "out/pagerank_data_" + str(i + 1)
    max_diff = "out/max_diff_" + str(i + 1)
    Pig.fs("rmr " + docs_out)
    Pig.fs("rmr " + max_diff)
    bound = P.bind()
    stats = bound.runSingle()
    if not stats.isSuccessful():
        raise 'failed'
    mdv = float(str(stats.result("max_diff").iterator().next().get(0)))
    print "max_diff_value = " + str(mdv)
    if mdv < 0.01:
        print "done at iteration " + str(i)
        break
    docs_in = docs_out

#!/usr/bin/python
from org.apache.pig.scripting import *
pig = Pig.compile("""
    input = load '$country' using CensusLoader();
    ...
    store output into '$country_out';
""")

    params = [{'country': 'Afghanistan', 'country_out': 'af.out'},
    ...
              {'country': 'Zimbabwe', 'country_out': 'zw.out'}]

    bound = pig.bind(params)
    stats = bound.run()

Once we have our BoundScript object we can call runSingle to run it. This tells Pig to run a single Pig Latin script. This is appropriate when you have bound your script to just one set of variables. runSingle returns a PigStats. This object allows you to get your results, examine what happened in your script, including status, error codes and messages if there was an error, as well as getting statistics about the run itself. Table Table 9.1, “PigStats Methods” summarizes the more important methods available for PigStats.

As seen in the example the OutputStats object returned by result() can be used to get an iterator on the result set. With this you can iterate through the tuples of your data processing result in your Python script. Standard Tuple methods such as get() can be used to inspect the contents of each record. See the section called “Interacting with Pig Values” for a discussion of working with Tuples. Based on the results read in the iterator your Python script can decide whether to cease iteration and declare success, raise an error, or continue with another iteration.

In addition to the compile, bind, and run methods presented so far there are also utility methods provided by Pig and BoundScript.

File system operations can be done by calling the static method Pig.fs. The string passed to it should be a valid string for use in the Grunt shell. See Chapter 3, Grunt. The return code from running the shell command will be returned.

You can use register, define, and set in your compiled Pig Latin statements as in non-embedded Pig Latin. However, you may have a jar you wish to register, a function alias to define, or a value to set that you want to be effective through all your Pig Latin code blocks. In this case you can use the static methods of Pig described in table Table 9.2, “Pig Utility Methods”. The registers, defines, and sets done by these methods will affect all Pig Latin code compiled after they are called.

# register etc. will not effect this block
p1 = Pig.compile("...")

Pig.registerJar("acme.jar")
Pig.registerUDF("acme_python.py", "acme")
Pig.define("d_to_e", "com.acme.financial.CurrencyConverter('dollar', 'euro'"))
Pig.set("default_parallel", "100")

# register etc. will effect p2 and p3
p2 = Pig.compile("...")
p3 = Pig.compile("...")

Once a script has been bound and a BoundScript returned, in addition to running the script, you can also call describe, explain, or illustrate. These do exactly what they would if it were a non-embedded Pig Latin script. However, they do not return the resulting output to your script; it is dumped to the standard out. (These operators are intended for use in debugging rather than for returning data directly to your script.)