LOGIN 2009: Middleware Roundup

Moderator: Joe Ludwig (JL)
Patricio Jutard (PJ), Three Melons
Chris Dyl (CD), Turbine
Rick Lambright (RL), Gazillion
Mitch Ferguson (MF), Carbine

Summary

A lot of nice little anecdotes about MMO middleware experiences – but it would have been nicer to have had a lot more detail on them (not easy to do in panel format).

The panellists briefly touched upon several few sacred cows – such as “Is it absolutely essential to have the full source code to exernally-licensed middleware?” (which I’m glad to hear being questioned – with server middleware especially the developers are hopelessly underqualified to understand huge chunks of source, even at a debugging level (e.g. the many cases where the devs simply cant practically debug distributed-system source)). There are good reasons for having source (and the other sacred cows) but more questioning and more critical thinking from developers about what they *really* need and especially what they’re going to *do* with it would be good for everyone, I think.

Any mistakes understandings and misattributions etc my own fault. My personal comments in [square brackets]

Talk

If you were going to make a new engine/MMO today, what middleware would you use?

PJ: we focussed on web based games when we started. We decided to use Unity, no royalty sharing made it a very attractive pricing model.

CD: hypothetically … I’d make substantial use of middleware, although we didn’t really at Turbine because it wasn’t there when we started.

RL: we look at what’s going to be unique about the game design we’re starting, and then look for middleware that matches well with the uniqueness of the individual game, whatever gets us a good chunk of the way towards that.

MF: we’ve decided each time to just do it ourselves.

How do you deal with the risk of using middleware code that’s never been used in a shipped MMO, that you’re the first major user of / the other major users haven’t shipped yet?

PJ: make sure you have the source code (although we couldn’t afford the more-expensive source-code license for Unity), make sure you have good support and responsiveness from vendor Tech Support.

CD: make sure the vendor is committed to the vertical business space, that your success is part of their core product strategy

JL: being first shipping user can get you a big price discount from the vendor

RL: with BigWorld we don’t have all the source. You have a limited number of engineers who’ll be skilled enough to do anything meaningful with that source anyway, and they could quit, leaving you in trouble. We have a commitment that if something exceptional comes up then we have the *possibility* of the vendor giving us source access temporarily, with their engineers physically present at all times, but we hope not to need that.

MF: any middleware that includes game design / game-logic elements is highest risk. I’ve had low-risk pieces of middleware fail, but that’s quite easy to workaround because it’s got relatively little effect on the overall project. I don’t want to name vendors in particular examples, because it’s not necesssarily their fault.

JL: we used SpeedTree for a couple of years, but eventually didn’t ship with it, because it couldn’t integrate with our art pipeline. Not a bug in that, just unfortunate accident of how it works and how we worked.

CD: we had the same problem with ST.

CD: the two scariest middlewares we used at Turbine were having Physics, and Kynapse for path-planning. Do we de-stabilise our live servers by integrating this cutting edge middleware new version in? As an MMO operator you really want to be able to keep an old version in situ for a long time – maybe a lot longer than the middleware vendor actually wants you to have it there.

[ADAM: that’s an extremely interesting problem…]

We had a big problem with a middleware engine where the provider just went out of business. We got some support from our publisher there to work through it, but what do you do about that?

CD: source code escrow is a good thing, for the absolute disaster scenario. But it’s really no different from other catastrophic business risks. Make sure you have one (more than one) person who’s been over the source in detail with the vendor at least once, so understands it well enough to pick up if they vanish, etc.

RL: escrow is mandatory, including all the required libraries, correct verisons etc.

[ADAM: nb: same speaker who doesnt have full source for some aspects of BigWorld]

JL: we didn’t escrow Alchemy, too naive to realise we should, but we got someone to send it to us as they were going out of business. Just getting it to build was a huge problem, we found.

[ADAM: our industry is still cursed by the fact that almost no-one has a great build system in place :) ]

MF: with Renderware we had source for the main thing, but there were dev tools that we had no source for, and so we couldn’t fix bugs in the tools, it was a nightmare. Very fine Modularity of your middleware – which doesn’t really exist yet – would be a real saver in this area. A hyper-modular middlware, I think that would make the risk much lower. Then if the company did go out of business, you could replace those building-blocks one at a time. I think that modularity is very important, or as close as you can get.

[ADAM: ha. I fought hard at the tech level to keep exactly that as one of the top 3 features of our server middleware back in 2002, but unfortunately back then (my naivety in assuming developers understood more than they did) no game developer had anywhere near enough understanding of the problems to realise how valuable that was. Ultimately, it was a waste of money for us as a company, because no customer understood it enough to place the value on it that would pay for how much it was costing us to achieve and maintain it. I’m not convinced that’s changed enough to make it profitable enough for MMO providers – even if developers now see the theoretical value, I don’t think they are able to really afford it]

Audience: Given how heavy the sales process is … is there a role for open source (OSS) in this?

MF: I think OSS is actually necessary for our industry to remain profitable, to reduce the cost of development. If I say bad things about middleware it’s not because I want it to fail (I really want it to succeed, I think it’s critical for us), but because I’m so disappointed with the current state of it.

RL: how many games have Python in them? Well, that’s an example of OSS that’s now heavily in use. We have an enormous amount of OSS we use inside the backend systems / servers.

What are the packages are you using for service side?

CD: we use a 3rd party billing system, that I wouldn’t actually recommend. We chose it many years ago, and it isn’t really right for what we do now.

(audience): there’s big advnatages to using external billing provider, e.g. benefit of having someone else as the merchant of record, isn’t there.

[ADAM: personally, I think that’s short-sighted, because in many ways the last thing in the world you want to have happen, as a service provider, is be disengaged from the customer’s wallet. Never let someone else come between “you” and “your customer’s money”; they can handle it, facilitate it, but not take over the CRM]

What about internal middleware? A lot of companies form Core Technology teams, creating a culture of making MW for internal use, re-use in other titles, etc

CD: D&D and LoTRO have the exact same codebase on the platform level because of our internal middleware. We ported the LoTRO DirectX 10 layer as a full working initial Dx10 directly into D&D, without changing it.

RL: the question comes down to whether you have a very specific target for what you’re doing. If you try to make one engine for all your games, and/or all your studios (if you’re multi-studio) then that’s a route to huge difficulties and disasters. At the same time I’ve seen companies that have huge in-house expertise in their internal wide-use engine and are able to churn out new games easily. But I think it’s a really really difficult problem to solve that way.

CD: we can have a full game prototype from zero to running in 3 months because we have all this internal re-usable code. For a relatively low investment we can build a small portion of a world to try out new game concepts.

[ADAM: I’m guessing the caveats there are simple that the content-cretion costs are of course still directly proportional to how much you want to show – hence “build a SMALL portion” etc]

MF: It was painful sometimes when the company was going down that they weren’t helping on the games themselves directly, but when Perpetual folded, the core tech group, the reusable code, was the one part that they managed to sell off. Clearly it has a lot of value, even if it’s painful getting there.

PJ: I think it makes sense to have one team on gameplay and building things, and one focussed on technology, because it’s different expertise for the staff to have.

What about MMO Middleware on consoles? Will midleware bring MMOs to consoles finally?

CD: it’s absolutely helped the transition, you’ll see stuff very soon where it’s facilitated console MMOs.

RL: stuff that’s provided to you multi-platform is a huge win from using middleware.

Questions from Audience

Q: sounds from what you’ve said here today like the state of the middleware industry today is fragile and embryonic?

MF: I agree. I think it’s a difficult industry. Business model is tough, it’s sales heavy, hard to get your cash in when people go under –

JL: – that’s what happened with Intrinsic, they had agreements with lots of companies paying a small amount each month towards the license fee, and when a few of those went out of business, the sudden cashflow damage killed Intrinsic

Q: what are your experiences with the multi-platform engines?

RJ: we tried most/all of the flash stuff. We’ve found Unity to provide the most-productive pipeline, fast and easy to prototype with, works nicely with our dev methodology (scrum).

RL: the more MMO development you do, the more you realise it’s all about the tools. The content-creation tools etc become more important, ultimately, than the engine itself.

CD: the cost of developing an MMO is never the technology, four or five times that cost is the cost of the content-generation. So the more efficient your tools are, the quicker you can get final game-content being made, that’s key.

Q: (from MF) does middleware “help” or “hurt” innovation?

MF: I feel it could hurt quite a lot. e.g. Some years ago, I could not find any middleware that would support instances, so we couldn’t use it.

CD: agreed. I think something monolothic like BigWorld could be particularly limiting, but the piecemeal approach of lots of little parts, increased productivity, actually I think helps innovation because it helps on your dev-resource costs a lot.

Q: are any of your perspectives on middleware non-technology centric, it’s been very tech-centric so far? I’ve seen a lot of recurring frustration with middleware. What kind of heuristics do you use to establish whether middleware is the right choice?

RL: you should never take a fully-staffed team and then dump in middleware on them. The two should be built hand-in-hand; middleware changes your hiring needs, and vice versa.

CD: Tech is always both an enabler and a limiter.

Q: as a vendor looking-in … we see particular teams (art vs code vs design) a lot more vocal than others, and the decision to use/not-use it gets skewed.

RL: the purchase decision is often lead by the lead client engineer, server stuff gets ignored, which is clearly wrong.