Five Minutes on Perl and Google

Introduction

• What? Accessing Google programmatically
• Why? You want to do search the web using a perl script
• When? Any time
• Where? http://www.google.com/apis/
• Who? You?
• How?
  • Get a free alphanumeric key from Google
  • Free download of WSDL file and some other stuff (mostly reams of documentation)
  • Use a perl module or two (Soap::Lite, Net::Google)

Modules

Barebones Way

• use SOAP::Lite. Need 0.52 or greater.
• You get a WSDL (Web Services Description Language) file
• XML definition of function arguments and return values)
• Example of the ResultElement

  
  <xsd:complexType name="ResultElement">
   <xsd:all>
    <xsd:element name="summary" type="xsd:string"/>
    <xsd:element name="URL" type="xsd:string"/>
    <xsd:element name="snippet" type="xsd:string"/>
    <xsd:element name="title" type="xsd:string"/>
    <xsd:element name="cachedSize" type="xsd:string"/>
    <xsd:element name="relatedInformationPresent" type="xsd:boolean"/>
    <xsd:element name="hostName" type="xsd:string"/>
    <xsd:element name="directoryCategory"
  type="typens:DirectoryCategory"/>
    <xsd:element name="directoryTitle" type="xsd:string"/>
   </xsd:all>
  </xsd:complexType>
  
  

• Back in perl, this maps to a hash, which you would access with something like:

  
  use constant LOCAL_GOOGLE_KEY => "********************************";
  my $googleSearch = SOAP::Lite ->service("file:\\googleapi\\GoogleSearch.wsdl");
  my $query = 'hello world';
  my $result = $googleSearch -> doGoogleSearch(LOCAL_GOOGLE_KEY, $query, 0,
  10, "false","", "false", "", "latin1", "latin1");
  
  for (@{$result->{"resultElements"}}) {
  	print $_->{"URL"}."\n";
  }
  
  

Easy Way

• use Net::Google, which gives a nice OO interface.
• Sample code:

  use Net::Google;
  use constant LOCAL_GOOGLE_KEY => "********************************";
  
  my $google = Net::Google->new(key=>LOCAL_GOOGLE_KEY);
  my $search = $google->search();
  my $query = 'hello world';
  $search->query($query);
  
  for (@{$search->results()}) {
  	print $_->URL()."\n";
  }
  
  
  

Features

• Use Google cache
• Spell-checking
• Use full range of Google's filtering features and advanced search criteria

Sample Project

• Personal project I'm working on is a plagiarism detector for Wikipedia
• Connects to either random pages or a pre-determined list. Grabs an article from Wikipedia using LWP, uses Google API to see if there are existing files with the same phrase(s) as that article. Optionally spiders remote page to do a more extensive check.

Advantages

• Avoid screen-scraping or trying to decipher querystring parameters
• API should still work even if Google changes things around
• Fast & Easy to program

Disadvantages

• 1000 queries a day is not all that much
• Can only view ten results at a time
• Limited to 10 search terms (same as through web interface)
• Beta service

Other Resources

• New book from O'Reilly, "Google Hacks", contains 100 different tricks and tips.
• The author, Tara Calishain, has a web site: http://www.buzztoolbox.com/google/ with some interactive stuff.
• Google Web API documentation: http://www.google.com/apis/
• google.public.web-apis newsgroup

These are some notes for a quick presentation I did for Ottawa Perl Mongers as part of a roundtable session we had on March 12, 2003.

Home | My Blog