Mike On Ads

Sorry for a bit of self promotion, but, Stephanie Clifford from the New York Times just published a great story on real-time bidding and how AppNexus & eBay have worked together over the past year to dramatically increase ROI with Real-Time buying from companies like Google:
Instant Ads Set the Pace on the Web.

Scary, I’ve been trying to explain to my mother what I do for years, but I think she’ll finally get it after reading this.

Apologies to all that posted comments on the blog the past 2 weeks, appears that my comment spam filter has marked every comment submitted as spam! I went through a good chunk just now but have about 1500 spam comments to weed through so there are probably a few I’ve missed.

I still have to figure out why the spam filter is broken, but please be aware that comments might take a few hours to show up but aren’t going into a black hole!

-Mike

PS: Normally comments are auto-approved unless they contain too many links …

Unless you’ve been living under a rock you’ve heard the term ‘DSP’ — “Demand Side Platform” thrown around like no other. The term has hit such a buzz that there is almost no meeting that doesn’t start with — “Wait, are you a DSP? What is a DSP? Is ____ a DSP? Are you working with a DSP? Which DSP should I work with?”.

Let’s start with some history. I’m not quite sure who coined the term originally but it was primarily used to describe cross exchange buying desks & bid management solutions like Invite, Turn & DataXu. We at AppNexus even briefly described ourselves as a DSP — that was of course when the ‘Platform’ was still a part of DSP — more on that in a bit.

Since then, agencies started allocating budgets away from “networks” and towards “DSPs”…

Wait… what?

You got it –agencies, eager to cut into the hefty margin networks take, started to allocate budgets towards DSPs for exchange buying. What’s of course ironic, is that the typical relationship between an agency and a DSP is certainly not a “platform” relationship but a simple media IO. Early versions of the platforms weren’t (and many still aren’t) mature and the only way they could make them work is by having people manually traffic and optimize campaigns on ad exchanges. Although the spirit of the relationship was one of a platform optimizing media across exchanges the reality is that it is primarily a service driven offering… something which practically looks very similar to a network with a slightly more defined box of inventory and in some cases clearer rules & margins.

Networks of course become massively threatened when their IO budgets got cut in favor of DSPs & Exchange Buying. Now if you’re a network, what do you do? You just rebrand yourself as a DSP or launch a new DSP arm of the business! Voila… now you can go right back to the exchange and say — “Oh, we do that too!”. Now every ad-network is suddenly a DSP…. Of course the response from the first set of DSPs has been to quickly try to define DSPs as “not a network”. You can read two examples on AdExchanger here: Zach Coelius @ Triggit, Nat Turner @ Invite.

Now the majors (Google/Yahoo/MSFT) of course see their direct relationships with the agencies being marginalized by this new “DSP” concept. So what do you do… that’s right, you become a DSP!! Yahoo has already announced this strategy and Google has been rumored to buy a DSP — Microsoft can’t be far behind.

There you have it… who is a DSP? Everybody. The problem is that in this whole process the ‘P’ of ‘DSP’ has disappeared — where is the platform– it just seems to be anybody who plays on the Demand Side — technology vendors, ad-networks & brokers, portals and a few hybrid tech/network companies. Platforms draw valuations, therefore everybody is a platform. The thing is that platform implies that people can build technology on top of your technology. Of the umpteen companies calling themselves a ‘DSP’, how many can say that there is technology being built on top of them? How many have open APIs that you can integrate with?

Here is my proposal… let’s retire the term DSP. It’s loaded, and effectively doesn’t mean anything anymore. Instead, let’s talk about Display Engine Marketing (DEM) and ad technology vendors. DEM companies are ones that will take your media $ and optimize it for you across aggregators of Display. This is the media relationship. Adserving & RTB technology vendors are ones that will license a technology — which may or may not be a platform on it’s own — to integrate with supply aggregators and help run a DEM business. Of course going to be DEM companies that build their own and some that license others, that’s expected in this world. Similarly there are going to be technology vendors that have their own in house DEM teams (*cough* dem==ad network *cough*) that will take an IO and run the media on your behalf.

In the end I want to point you back to an old post I wrote: I don’t care who you say you are, what do you DO?. The next time someone says they are a DSP — respond with — “That’s great, but what do you actually do?”

Cookie Matching.

A number of people have asked about how cookie matching works in the RTB world. Everybody today relies on cookies to store information about a user — which ads he’s seen, what sites he’s been to, which behavioral segment he’s in — all the good stuff. Without cookies, we can’t frequency cap, we can’t remarket, we can’t track, basically, advertising becomes pretty useless!

Now the way browsers work, cookies are tied to a domain. Say an ad-exchange has ad tags on a page under ‘ad.exchange.com’. When a browser requests an ad from the exchange, it only passes along the cookie data that’s stored inside that domain name. This means that the exchange has zero insight into whatever data the bidder might have collected under ad.bidder.com.

So why is this a problem? Well, if the exchange is going to do a server-side bid request to the bidder, it’s pretty hard for the bidder to make a decision about what ad to serve if it has no access to either frequency or behavioral data!

An Example. Bidder is running a behavioral campaign for marketer “marketer.com”. Bidder has a pixel on the marketer’s site, it’s internal ID for the pixel is 27. Each time this pixel fires the bidder ads a cookie to it’s domain ‘ad.bidder.com’, ’segment=27′. The bidder now wants to buy as much inventory as possible for this remarketing segment across exchanges. Bidder has a user-id, ‘ABC’ for this cookie.

Exchange, ‘exchange.com’ wants the bidder to be able to buy as much inventory as possible. Hence the exchange wants the bidder to be able to use all remarketing segments to buy. Problem is (as mentioned above), they’re using separate cookie domains. So what needs to happen is that the exchange and the bidder need to synchronize their domains so that when the bidder receives a real time bid request he can look-up the information associated with that user.

Fundamentally the logic for synchronizing IDs is pretty simple. The exchange drops a pixel on a page somewhere which points to a bidder’s domain and a simple usersync call. When the call hits the bidder, it reads in it’s cookie ID, redirects off to the exchange passing it’s ID in the querystring so that the exchange can read and store it into it’s cookie space.

The above diagram shows this process. First a pixel is dropped that hits the bidder domain. The bidder receives this call and reads in it’s cookie ID of ‘ABC’. Bidder then redirects the user to get a request from the exchange under ad.exchange.com, passing in the id ‘ABC’ in the querystring. The exchange now receives a call, reads in from the cookie that his ID is 123, and can now map that his user 123 maps to bidder’s user ABC.

Now user IDs are synchronized, the exchange and the bidder can talk about a single user in the same manner. Note that in the above example I noted that the exchange is storing the ID for the bidder. Admeld, PubMatic & the like all work like this. Google does it slightly differently and instead requires the bidder to store the mapping on his end — effectively just the reverse of the above process.

One of my readers posted the following comment on my first post on RTB:

In your second diagram you show the interaction between the publisher adserver and multiple networks. Does this potentially multiple source back and forth not slow down the adserving in the same way a series of dumb redirects would? Especially when you consider that presumably if Network 1 came back with the best price out of 3 or four networks, once the publisher ad server knew that it would need to go back to it and request the actual ad again. It would be interesting to see some realistic HTTP traces for this stuff.

This is indeed a great question. Technically it looks like there are the same # of requests going back and forth in RTB versus a traditional ad-call. Although this is the case, RTB is going to be significantly faster… and here’s why.

Technically a browser downloading content from an adserver is a five step process:
* DNS lookup of the adserver domain name
* Establishment of a TCP connection
* Requesting content
* Acknowledge of request & sending back content
* Terminating the TCP connection

Assume for this case that a DNS lookup takes about 100ms. Each of these steps requires a number of packets to go from the local computer up to the adserver and a series of response packets. Here’s the # required for each step:

* TCP Connection: Two packets up, and one packet down (SYN, SYN-ACK, AKC)
* Requesting content: One packet up (minimum)
* Request acknowledgement and content: One packet down (minimum) & one packet up
* Terminating the connection: One packet

So the minimum number of packets sent back and forth is 7. If the latency from an end-user is 50ms to the adserver, this means it will take *at least* 450ms (100ms DNS + 350ms ad-request) to request the ad.

Now you’d think this would be the same for real-time, but it’s not! There are three reasons a request between two serving systems is much faster:
* Better connectivity — Adservers are hosted in datacenter that generally have much better internet connectivity than the average end-user. This means lower latency between the two adserving systems.
* No DNS lookup — The RTB system can cache DNS lookups for all RTB partners, effectively removing this 100ms.
* Persistent TCP connections — Any intelligent RTB integration would use persistent TCP sessions between the sell and buy side systems. This means a connection is established once and reused thousands of times after that.

With the above three, here’s how requesting a “bid” looks from sell to buy side:
* Requesting content: One packet
* Acknowledge of request & sending back content: One packet

So assume 25ms latency between systems (rather than 50) and the minimum time for an RTB request between systems is only 50ms compared to the 450ms it would take for an actual end-user or 9 times faster. The slower the end users connection and the faster RTB will be.

Conclusion — yes, adserving individual requests becomes a little bit slower but the removal of redirects makes the overall process signficantly faster.

For those technically curious, here’s are tcpdumps that prove this.

Browser to adserver:

15:45:04.380042 IP 10.0.1.31.59541 > 8.12.226.77.http: Flags [S], seq 50484529, win 65535, [...]
15:45:04.397395 IP 8.12.226.77.http > 10.0.1.31.59541: Flags [S.], seq 661028066, ack 50484530 [...]
15:45:04.397529 IP 10.0.1.31.59541 > 8.12.226.77.http: Flags [.], ack 1, win 65535, length 0
15:45:04.397831 IP 10.0.1.31.59541 > 8.12.226.77.http: Flags [P.], seq 1:1288, ack 1, win 65535, length 1287
15:45:04.424466 IP 8.12.226.77.http > 10.0.1.31.59541: Flags [.], seq 1:1461, ack 1288, win 62780, length 1460
15:45:04.424472 IP 8.12.226.77.http > 10.0.1.31.59541: Flags [P.], seq 1461:1543, ack 1288, win 62780, length 82
15:45:04.424546 IP 10.0.1.31.59541 > 8.12.226.77.http: Flags [.], ack 1543, win 65535, length 0

Adserver to adserver with persistent connections:

20:00:10.709152 IP 64.208.137.8.41096 > 8.12.226.77.80: . ack 1023 win 7154
20:00:10.754844 IP 8.12.226.77.80 > 64.208.137.8.41096: P 1023:2045(1022) ack 501 win 62780

Check out the post by Tim Cranton on theMicrosoft Blog about five suits he just filed against a variety of malware purveyors.

Kudos to Microsoft for taking action! The wild-wild west of display advertising is slowly growing up.

Please check out my last post on RTB first. Since this last post, a pretty big announcements has hit the wire. Namely, Google has announced Ad Exchange 2.0. Most significantly:

One of the platform’s key features is the ability for ad networks and agency buyers to bid on inventory in real-time, letting them zero in on impression attributes such as geographic location or the presence of advertiser cookies before placing a bid. Yahoo’s Right Media ad exchange does not currently offer bidding in real-time, though it is available through some smaller ad marketplaces.

That’s right folks — Google’s real time exchange is coming. In this post we’ll talk about who is jumping on the RTB band-wagon on the supply side, and some implications this is going to have on the industry.

Ok Who’s In

Ok, so Google is doing it? Who else? Over the past few months pretty much any aggregator of supply has launched, announced or started work on some sort of RTB capability. All major exchanges — Yahoo’s Right Media, Microsoft’s AdECN and Google’s AdEx have RTB integrations in the works. Of the pub aggregators, AdMeld & PubMatic are live and Rubicon is actively working on a solution. As mentioned, FAN has been live with Myspace inventory for a while and there are a number of other parties, such as ContextWeb, AdBrite and OpenX, entering the space. The short summary is, over the next 12 months we can expect billions of daily impressions hitting hundreds of millions of unique users to become available RTB.

Death of the Traditional Ad Network

This is going to have huge implications for the display advertising space — primarily, the traditional ad-network model is on it’s last legs. Most traditional ad-networks today thrive because they have large business development teams that have developed deep relationships with supply. Ad.com, Specific Media & ValueClick all have large publisher bases that they rep to agencies. This posed a large difficult hurdle for new networks to overcome and effectively created a catch-22 for any new network entering the space. To get media dollars a network needs reach, but to get publisher deals a network needs media dollars.

This all changes when there are billions of biddable impressions out there. In this new world, any new network has instant access to the reach that historically would take years to build up. Now anybody can walk into an agency and claim to have a reach of over 100 million unique users.

Now of course this isn’t totally new. Right Media opening up Yahoo inventory to the world back in 2007 started this process and a number of companies have managed to start very succesful network (or “exchange desk”) businesses on this platform. AdECN, AdEx, Admeld, PubMatic and Rubicon take this to a new level as this opens up MSN, the DFP publisher base and the majority of the Comscore 1000 list!

Can Technology Finally Win?

I’ve written in the past about the plight of the ad-technology startup. The short summary is this — technology is great, but the lions share of revenue today comes from media not tech.

As access to inventory becomes commodity marketers and brands will inevitably start focusing on results over the ability to simply spend the budget. With everyone on an even playing field it’ll be easy for a marketer to compare the results from one buying network to the next — which means technology will finally matter.

There are already a number of successful technology focused startups who focus on exchange buying — and a couple that are simply building cross-exchange buying tools. Expect this to become the next hot space for startups in the advertising world.

Behavior behavior behavior!

The traditional problem with behavioral buying, whether remarketing or using third party data provided by companies like IXI, Exelate or Blue Kai, has always been reach. We all know that remarketing works wonders and has amazing ROI, but unless you can actually find your users on the web it’s hard to spend a significant amount of money this way.

With billions of impressions that of course changes — the probability of finding a user across the many RTB platforms becomes easier and easier and hence the actual required reach of any given behavioral segment becomes smaller and smaller. This in itself is going to make data-focused businesses more feasible and also open up a world of possibilities of highly targeted and focused media-campaigns and very granular behaviors.

Next Up

Ok, enough RTB for the day. Next up –> demand demand demand! Who are the new players that are taking advantage of this new RTB revolution and innovating both from a business model & technology perspective.

Ok, so in the last post I explained what Real Time Bidding (RTB) was. Greg Yardley wrote a response on his blog which made me realize that I didn’t spend quite enough time on the technical implications of RTB. He wrote:

[...] If you’re using real-time bidding, and you’re buying one impression out of a hundred, it’s got to pay the cost of the ninety-nine other decisions not to buy. [...] I think you’ve got to have as close to real-time reporting as possible. [...] Finally, your models have got to account for the data that can’t arrive in real-time, or you’re going to put your efficiency gains at risk. [...]

Add all of the above up, and you may be wishing you stuck with your predefined rule sets.

So I guess I should clarify — building a real time bidder is technically almost exactly as complex as building a performance aware ad-server — it needs all the same pieces:
* Decisioning algorithms
* Targeting & frequency tracking
* Server-side cookie store
* Log Collection & Aggregation Pipelines
* Reporting & Analytics
* Budgeting & Campaign Pacing
* A Trafficking interface

This is certainly not an undertaking to take lightly. Many developer years go into building a scalable serving platform — so yes, hiring the talent and building out the infrastructure to build a scalable bidder is certainly an expensive proposition. But also, note that if you already have an adserver, it likely has a lot of the features listed above.

Put it another way — Real Time Bidding allows you to plugin your intelligence into a massive pool of inventory. If you don’t have any intelligence yet, obviously that will be a significant investment! I’ll talk more about building “bidders” in future posts.

Now to continue the intro on RTB.

A little aside from the personal blogging — I’m extremely excited to announce that Michael has joined the AppNexus executive team! Press release below:

AppNexus Names Google/DoubleClick Ad Exchange Executive as President

Company Prepares for Growth in Emerging Fields of Cloud Computing and Real-Time Advertising

NEW YORK, NY – September 9, 2009 – AppNexus, Inc., a cloud computing and real-time online advertising platform company, today announced that Michael Rubenstein has been named the company’s President, reporting to CEO and co-founder Brian O’Kelley.

Mr. Rubenstein is a 10-year veteran of Google/DoubleClick, where he has served in a variety of business leadership and executive management roles. Mr. Rubenstein most recently served as Google’s Director of Ad Exchange. Prior to Google’s acquisition of DoubleClick, Mr. Rubenstein founded and served as Vice President & General Manager of DoubleClick Ad Exchange, reporting to the company’s CEO David Rosenblatt.

“We are delighted to welcome someone of Michael’s extensive online experience, vision, and record of accomplishment to the AppNexus team,” said Brian O’Kelley, CEO of AppNexus. “AppNexus’s business is at the intersection of the most exciting emerging segments of the internet, including cloud computing and real-time online advertising, and Michael’s experience successfully commercializing new offerings will be an invaluable asset to the company as we continue our growth . This appointment is an important step in the company’s ongoing expansion and underscores our commitment to adding top industry talent.”

About AppNexus, Inc.
AppNexus is a cloud computing and real-time online advertising platform company based in New York City. AppNexus’s customers and partners include the largest buyers and sellers of online advertising. AppNexus’s proprietary technology platforms and cloud infrastructure are powering the development of the real-time advertising ecosystem. The company was founded in 2007 by Brian O’Kelley and Mike Nolet, and is backed by investors including Venrock, Kodiak Venture Partners, and First Round Capital. More information can be obtained by visiting AppNexus online at www.appnexus.com.

Checking my news this morning I stumbled upon This Silicon Alley Insider post. The post has a simple AJAX application inside of it which lets people page through the “Top 10 iPhone Apps for Students”:

What’s interesting about this application is that every time you page left or right it refreshes both the 728×90 ad unit at the top and the 300×250 on the right — with no timer. Try it… in the span of 2 seconds I managed to get 20 ad views on this page — almost all for American Express and IBM. Complex AJAX applications pose an interesting challenge for publishers — think a live flash based stock ticker or Pandora — if someone is staring at a web page all day but doesn’t actually navigate away, it certainly doesn’t make sense to only show them a single ad. This on the other-hand seems to be taking that to another extreme. I mean, come on, most blogs will simply post a list of 10 applications and not a silly widget which appears to only be there to generate ad-views. Here’s a little snapshot from my HTTP sniffer:

Now the IAB recently published guidelines to specifically address this issue in their IAB Rich Internet Application Guidelines. I thought I’d go check these out, find a quote that specifically addressed SIA from showing 20 ads within just a few seconds. Guess what… technically SIA is following the rules:

User activity considered significant enough to trigger ad-counting is as follows:

Mouse Button Usage
Keyboard Activity, Typing (except when used to navigate away from the page alt-tab, etc).

This counting method closely resembles the current IAB Ad-Impression counting guideline because of the clear linkage of certain pre-identified user-initiated events with ad counting.

This highlights a typical problem with IAB guidelines. There are so many parties voicing opinions in the working groups that they are made to be so broad that following them doesn’t really mean anything.

Having my eye on dynamic applications today I noticed that Facebook has implemented ad refreshes while browsing photo pages. This makes a lot of sense to me, users can spend a significant amount of time browsing photos and it certainly makes sense to refresh the ad-content after a certain amount of time. It looks like they’ve set a 60 second timer, and only refresh ads if a user actually clicks through to a new photo after that timer has expired — much better!