projects / Gmail Agent API / Mail Notifier & Address Importer

Final update: The Gmail infrastucture has evolved quite a bit over the last decade so this information and accompanying software are largely obsolete. Leaving this here for posterity's sake.

Update 5/18/2005: The long awaited SourceForge project for this API has been created. Eric Larson has been gracious enough to help out with the project. See http://sourceforge.net/projects/gmail-api for the latest release.

What is it?

There are two distinct components here: an open source Gmail API written for the .NET framework, and a proof of concept Windows application built on top of that API that provides basic remote Gmail functions.

I developed these tools in the hopes of encouraging others to create interesting Gmail services. Admittedly, this project may not have a very long shelf life, as Sergey has intimated possible mail forwarding and RSS support, not to mention Gmail's recent listing of upcoming features (Gmail login required) that estimates a slew of features that are listed as “working on it” or “we'll try”. The address book import is currently listed as “sometime soon” but it's actually available now in the Contacts window. As Sergey mentioned, an enterprise version of Gmail would be well received, and I have no doubt that there would an API to go along with that (I recently integrated a Google Search Appliance, and can attest to its extensibility). Whether or not Google is interested in pursuing such features for the public side remains to be seen. Nonetheless, I hope to keep this project going, and wouldn't mind joining the Gmail team — there are lots of features I'd like to see implemented in Gmail.

This project has been mentioned by InfoWorld, The Sydney Morning Herald, Google Hacks, and Adaptive Path.

Who's using it?

Other projects that are related to the Gmail Agent API:

About the Gmail Agent Applet

I'm sure most of you are more interested in the applet, so here are the features:

• Multiple account support
• Balloon notification of new messages with message preview
• Address book view and import (from tab-delimited text files)

The system requirements are:

• Windows 98SE, or higher
• Microsoft .NET Framework v1.1

I have implemented a couple recommendations to make this Mono compatible but the results are still spotty. Let me know if you've successfully debugged this under Mono. If you are interested in POP access to Gmail, check out GPopper (formerly known as Pop Goes the Gmail).

This is a proof of concept application, and there are plenty of idiosyncracies. It works great for me, but your results may vary. I welcome anyone who wants to contribute to polishing this app.

Download the installer »

About the Gmail Agent API

The goal of the API is to provide an extensible foundation for interfacing with Gmail. The objects in this namespace should be abstracted enough to be able to be adpated to any future changes Gmail makes.

Read the documentation for Gmail Agent API 0.6.1 to see what's available.

The main workhorse of this class is GmailAdapter. It is responsible for communicating with Gmail and maintaining the login and session information through the duration of the application. The GmailSession object holds all the state information for a single Gmail account, including a GmailThreadCollection of GmailThread objects. The GmailContact object represents a single Gmail address book entry. Again, GmailAdapter provides the methods to fill a GmailContact with information from Gmail.

Connection Overview

One oddity with the API is that it uses TLS instead of SSL for the encryption layer. For reasons unknown, the SSL provider was extremely intermittent and often failed to establish a secure link so I manually set the ServicePointManager to use TLS 1.0. This seems to be a common problem among .NET developers, and if anyone has a stable solution, I'd love to hear about it.

Here is a bare-bones example of how to establish a connection with Gmail using the API:

// init new adapter
GmailAdapter gmail = new GmailAdapter();

// create new session and assign username and password
GmailSession myAccount = new GmailSession();
myAccount.Username = "googler";
myAccount.Password = "showmethemoney";

// login and retrieve mailbox info
GmailAdapter.RequestResponseType loginResult = gmail.Refresh(myAccount);

// display mailbox info
if(loginResult == GmailAdapter.RequestResponseType.Success) {

    // show new inbox count
    Console.WriteLine("New Threads: " + myAccount.DefaultSearchCounts["Inbox"]);

    // if new threads exist, show the subject of the first one
    if(myAccount.UnreadThreads.Count > 0) {
        GmailThread newThread = (GmailThread)myAccount.UnreadThreads[0];
        Console.WriteLine("Latest thread subject: " + newThread.SubjectHtml);
    }
}

Source and binaries

Download the latest source code from the Gmail API SourceForge project page.

Download the Windows help file version of the API documentation.
The online HTML version is also available.

» Discuss the API and its applications over at the Gmail Agent Google Group.

All material released on this page relating to Gmail is licensed under the GNU General Public License.

About the Gmail engine and protocol

You've probably noticed that Gmail's interface is extremely fast when compared to other web-based email systems like Yahoo! Mail and Hotmail. This is a result of Gmail's placement of the UI engine on the client-side as a JavaScript module. Whenever you log in to Gmail, a copy of the UI engine is loaded into one of the HTML page frames and remains there for the duration of your session (credit has to be given to Oddpost for being the first ones who perfected this idea). Subsequent actions from the Gmail interface are then routed through the Gmail UI engine in your browser, which in turn makes HTTP requests (via the XmlHttpRequest object) to the Gmail server, interprets the DataPack (more on this later), and updates the UI dynamically. In contrast, Hotmail and Yahoo! Mail follow traditional web application models and reload the entire UI after almost every action.

The item most relevant to this project is what I refer to as the “DataPack”, a base HTML file that contains only JavaScript array declarations that the UI engine parses and then uses to determine what to update. The advantages of this should be immediately obvious: reduced traffic load, and increased functionality — especially for developers who no longer have to resort to crude “screen scraping” techniques to interface with web applications. Although the ideal situation for external developers would be an XML-based DataPack, the JavaScript version is sufficient (and I suspect it was chosen for performance reasons as well).

The DataPack format consists of individual “DataItems”, or JavaScript arrays wrapped in a envelope function. An example:

D(["ts",0,50,106,0,"Inbox","fd36721220",154]);

The function D() references a runtime evaluator within the Gmail engine, which then interprets the attached array parameters. The "ts" element indicates that this is a threadlist summary item, and the subsequent elements denote start index, threads per page, estimated total, threadlist title, threadlist timestamp, and total threads. This is the same format that is applied to all array parameters sent through the DataPack:

[<DataItem_name>(,<custom_attribute>)]

The mappings to all the DataItems can be found in the engine code source (/gmail?view=page&name=js). For instance, qu contains quota information, while ct contains category (a.k.a. labels) definitions. Read through that file if you really want to get everything you can out of Gmail.

Determining the right URL to retrieve the DataPack is pretty straightforward, as most requests will return the same basic information, such as quota, category count, and inbox count. The main thing that changes is the threadlist summary, which depends on what page you're looking at. All the main folders — inbox, starred, trash, spam, etc. — are all really just pre-defined searches within Gmail. For example, the inbox DataPack URL is:

/gmail?search=inbox&view=tl&start=0&init=1&zx=

The search query for all unread threads is:

/gmail?search=query&q=is%3Aunread&view=tl&start=0&init=1&zx=

The main parameters are search= and q=, which define what set of threads the user is requesting. The zx= parameter is a proxy cache defeater, and I've omitted it here for brevity. See GmailAdapter.MakeUniqueUrl() for more information.

Gmail exploits another advantage of the DataPack model to increase efficiency by allowing for an empty document. This is employed by the 2-minute auto-refresh request. The inbox URL adds a few more parameters:

/gmail?view=tl&search=inbox&start=0&tlt=fd8dfa2e31&fp=c155594240dcc7cb&auto=1&zx=

The tlt= parameter is the thread list timestamp, which is treated like a checksum in determining the state of the client versus the mailbox state on the server. If the client timestamp is older than the one on the server, then a full DataPack is sent. Otherwise, Gmail sends an essentially empty DataPack.

Revision History

Update 2/27/2005: The API is currently broken, due to Gmail's protocol change. I don't have time to work on the project, but I have received some fixes that I will review as soon as possible. -JH

v0.6.1 (10/24/04):Fixed some proxy issues, and updated API to recognize the "Drafts" folder.

v0.6.0 (9/6/04): Version 0.6 Released. This version adds proxy support, compiles under Mono (I think), and changes the default click behavior for the icon to open the Gmail site.

v0.5.1 (8/15/04): I've updated the API to work with the new login procedure. Download version 0.5.1 below. The changes were minor (Gmail moved the GV variable from a client-side Javascript call to a HTTP response header call), but a quirk with the .NET cookie handler was mucking things up. Apparently, sending a subdomained cookie (e.g. gmail.google.com instead of just .google.com) to the HttpWebRequest class causes it to report an invalid cookie domain. Bug or feature? You decide.

v0.5.0 (7/5/04): Initial version released.

Feedback

Head over to the Gmail Agent Google Group to discuss the project, or feel free to contact me directly with any questions.

All material released on this page relating to Gmail is licensed under the GNU General Public License. Everything else is released under a Creative Commons License. Gmail is a registered trademark of Google Inc.