FarmVille

The context

FarmVille was one of the largest live games of its era, a service measured in tens of millions of daily players, and a game that introduced a huge new audience to gaming. Zynga set the bar for running a live product: monitoring, release discipline, and experiment testing against a live game at that scale. I didn’t invent much of it. I ran it at the source, and years later I recognized the same practices in how Google ran its programs. That experience is what makes me a live-service specialist.

I joined Zynga after Sony and was able to leverage that experience, the QA methodology and release management discipline Sony holds its titles to, to quickly become a trusted producer and release manager for the studio. For my first 2–3 years I ran release management for every FarmVille release, producing for 2–3 dev pods, covering on-call emergencies after hours, and improving the on-call practices as I went. I was one of two producers when I joined; there were twelve when I left, most of them trained by me, and I finished as the most senior producer on FarmVille.

At peak the studio ran 120+ developers, more than 35 of them engineers, and my job was keeping all of that output on one cadence: new content landing for engagement and monetization, viral moments timed for growth, and production metrics healthy through it all. Two major releases a week, every week, into the live game.

What I owned

• Live release cadence. Planned and shipped multiple content drops a week plus platform updates, on schedule, to a player base in the tens of millions.
• Roadmap prioritization. Drove studio roadmap prioritization across live operations and new features, deciding what shipped now versus what waited.
• Release & QA process. Owned release and QA processes for both live and new games, and ran large-scale playtests to put real player feedback in front of design before features locked.
• Team leadership. Led a team of up to 12 producers, training and mentoring incoming producers as the studio scaled.

How a release shipped

Every pod kept a living branch of its own, mirrored against trunk, and every engineer worked in a dev branch off that. When a pod was up for the next release, its branch merged into trunk so downstream teams could integrate early, and we cut a release branch from there. The release branch went to a staging environment the ops team kept as close to production as they could, and once it built there the release was locked. After that only critical fixes or high-impact business items got in, and that call was mine, made with the lead engineer, QA, and product against priority, need, and risk.

Two content releases in a normal week, and as many as five in a heavy one. Each release had a named engineering owner who stayed accountable for its quality and fixes through launch. Getting an owner was not always simple. When a piece of work had no obvious home it came to me, and I negotiated it onto a release based on timeline, safety, and feasibility. If a pod would not take it, I escalated. When the calendar could not absorb it at all, we opened a new release branch and added a release to the schedule.

Not every release could be rolled back, so we classified each one in advance and put more people on the ones that could not. Features shipped dark behind flags and turned on by timer or by hand, ramped to a slice of the player base while product, engineering, ops, and QA watched. That gave us a softer option than a rollback: turn the feature off, or stop the ramp, fix, and resume.

I shared on-call ownership with the lead ops engineer. I was not an engineer and never owned the code. What I owned during an incident was the timeline, the communication going out to stakeholders, the incident report afterward, and the process change that kept the same thing from happening twice.

The content pipeline

One of my favorite projects at Zynga was a tool players never saw. FarmVille’s item data, every unique item that appeared in-game or in the store, lived in one giant XML file, and every release added 5–20 new items to it. With releases stacked back to back, that single file became a constant source of merge conflicts. Not every contributor was equipped to resolve them cleanly, and at our cadence it was expensive to keep qualified reviewers babysitting repetitive data merges.

The answer was a Content Management System (CMS) for FarmVille’s release content. The project was the CTO’s: he approved it and led the design, focusing on the auto-merge. Once I saw what it could do for the studio, I was on board and driven to make it work. I produced it, turned feature requests from the artists and designers who lived in the tool into the design, and helped improve the merge model itself:

• Automated, order-aware merging. Releases often carried upstream dependencies that hadn’t shipped yet. If Release A introduced a schema change and Release B was scheduled three days later, the tool automatically integrated A’s schema changes into the future releases that needed them, in the right order, so every dev build stayed clean and testable throughout development.
• Surface changes immediately. The original design merged upstream changes only when a developer chose to pick them up, so a schema change three releases ahead could stay hidden until it went live, then blindside a team mid-crunch. My rule: entering the tool means picking up every release ahead of yours. The tool knew the release order, so upstream changes were exposed immediately, never delayed.
• Two ways in, one safe pipeline. The UX met people where they were. Designers could author an item entirely through pickers and dropdowns; contributors comfortable with XML who only needed the auto-merging could import it directly. Both paths fed the same conflict-free pipeline, with bulk-input shortcuts for the large releases.

The tool was a great idea, but how we supported it made it a success. At launch it was too rudimentary. Three or four improvements later it had grown bloated and sluggish: it queried unnecessary data from the database before anyone needed it, and load times stretched long enough to interrupt people’s workflow. The fix came in two parts, smarter loading that fetched data only when the work called for it, and a major performance rework that included moving from MongoDB to a cloud storage model. I couldn’t tell you which of the two mattered more. Together they made it the tool the studio needed.

Rolling it out was its own education: teaching people, collecting feedback, and learning that producing software for your co-workers is a very different job from producing it for customers out in the world. Testing it went further still. We built a fake FarmVille release cycle inside the CMS test environment, a nested simulation of the real cadence, so the tool could be proven against realistic releases before it touched production. You learn a lot about the limits and constraints of software development building something like that.

The payoff: 90+ development days saved per quarter for a 100-person content team, and an entire category of merge-conflict risk and reviewer cost simply gone. At FarmVille’s scale, removing friction from the content pipeline compounded across every release that followed.

The outcome

At peak the service held 3M+ concurrent players, 36M daily and 100M+ monthly actives, 250M+ lifetime reach, and it generated over $1M in daily revenue. I am grateful for the experience and to have worked with so many great developers.