Unity: the “real” version 5 just released (#Unity3d 5.2 fixes many severe issues!)

A new version of Unity is now out – https://unity3d.com/unity/whats-new/unity-5.2 – and it’s a doozy.

I don’t normally blog a blow-by-blow account, but this set is particularly interesting.

It suggests the tech team has been focussed on fixing many “critical but un-sexy” problems. That is a noticeable change in direction from the past few years. Possibly this is a one-off to get Unity 5 working properly.

If you’re still using Unity 4.x, it is finally safe to move to Unity 5.

Unity 5 is not a complete replacement for Unity 4 – there are still major featuers of 4.x that have been removed from Unity 5 and have no upgrade-path – but it’s finally production-ready (or very nearly). That … is big news!

Hilighted improvements / fixes / changes

Please note: there will be a lot of negative comments here; that is inevitable, considering most of the change-list is “things that were broken, or wrongly implemented, or poorly designed – and are now not”.

To be clear: I’m pleased and excited by the overall shape of these changes.

Graphics: Added CullingGroup API, which allows you to specify a large set of bounding spheres that are integrated into the culling pipeline and subjected to frustum and occlusion culling

Unity has (amazingly) never had built-in culling accessible by developers, which is almost unheard-of in game-engines. This suggests we’re getting towards it (although spheres are only a first-approximation for culling, and I wonder how deep the interaction is with Unity’s pipeline here. I’d assumed we’d get access to something more like BSP)

Application.UnloadLevel. Unloading scenes is now possible while the game is running. References to lightmaps are cleared but you have to call Resources.UnloadUnusedAssets() to actually unload them.

Scenes in Unity are very much FUBAR as a concept – they were created by people who had very little understanding of computer-game design and development. Ideally, we’ll see Unity remove the whole “Scene” concept in a future major release, but in the meantime this is a significant step towards reducing their weaknesses, which is good to see.

Layout elements under a ScrollRect component may not work properly

No s***, Sherlock! Yeah, most people using Unity’s “new GUI” have found it’s fundamentally broken in a bunch of places. Good to see Unity working on this, and hilighting some of the issues – but really they need to create an entire webpage listing “the following parts are broken, don’t use them – or beware if you do”.

Right now, you tend to waste hours, half-days at a time … re-discovering known bugs in Unity’s GUI.

Scene view picking in linear color space does not work for DX11 when MSAA is used (quality: fantastic)

If you’re on Windows and have the highest quality settings, this is lethal: Unity simply doesn’t work. But it’s coming in an emergency patch very soon.

Mecanim: Playable API.

If this does what I think it does, then it was essential back when Mecanim was released (3 years ago); glad to see it’s finally here.

between 5% and 30% lower CPU cost for rendering

That’s an enormous speed-up for an engine as old as Unity – this is big news. Reading between the lines, the speed up is partly a side-effect of them making it a bit more multi-threaded (which became normal for game engines many years ago).

If we thought that was a lot, wait till you see:…

Particle rendering optimizations (especially for VR), with speed improvements varying from 15% to 50%

… 50% performance improvement! Wow. Name-checking VR here is probably the key item. We keep seeing VR’s extreme performance-requirements forcing engine-makers to up their game, and dig deep to find perf boosts. The “minimum” hardware reqs for basic VR rigs are very high even now, several years after Oculus Rift launched.

UI now uses multithreaded batching & geometry generation backend

This was the minimum we expected from the “new GUI” when it launched in Unity 4.6 almost a year ago. This fix brings the GUI most of the way towards being “production ready” … ironically, 6 months after Unity changed their website to hide the existence of the “old (but works) GUI” and pretend it didn’t exist.

Added parameter QueryTriggerInteraction to all physics queries (raycast, spherecast, overlap) to allow any query to override whether or not the query should hit Triggers

This is patching a MAJOR design bug in core Unity that has plagued Unity for many years. I want to play with this some more, and see how much we can un-**** Unity’s ****ed ray-casting system using it.

(I don’t believe any experienced game-developer would have accepted Unity’s original design here – it was clearly a terrible approach that would cause never-ending head-aches when making games. It mixed concepts that were very obviously unrelated and distinct)

Fixed Function shaders work on all platforms now (including consoles)

This one has been blogged about extensively by some of the graphics team, IIRC.

In my experience, Unity’s shaders have been OK, with a few that were terrible (e.g. all the Mobile ones), but the framework which holds them has generally been excellent: it’s isolated us from the pain and suffering of GPU-specific shader bugs. This house-cleaning doesn’t make much (if any) difference to most of us, but apparently keeps Unity’s internal code clean in a way that will facilitate rapid changes and improvements going forwards.

2D Rect Mask

TL, DR: “we finally implemented masking properly in the new GUI”.

See previous comments on the new GUI, and how it’s getting much closer to production-ready.

UI: Dropdown control

Yes, really – Unity has finally added the “dropdown”, which has been a global expectation in UI design for 20+ years (it was made compulsory in web-browsers back in the 1990’s).

Editor: Remove legacy Mac OS X corner window resize behaviour

Apple told devs to remove this … um … about 3 or 4 years ago? At first, it was broken (couldn’t resize windows), then it was annoying (Unity added optional resizing on all edges, but put an illegal resize-hot-corner in bottom right that broke all EditorWindow code, because they made it a private API).

Working around that on Mac … forced you to mildly break things on Windows.

TL, DR: custom editors (as found in many/most of the best Asset Store plugins) will now be a little less-broken across the board.

Substance … All crashes and hangs at build time, level load/change time, playmode entry/exit should be gone

Yay! I’ve not been using Substance, but I’ve met several developers who’ll be breaking over the champagne over this, given how miserable they’ve been about crashes and hangs with this in the past.

CanvasRenderer now takes a Mesh instead of List this allows for the use of imported meshs as part of the UI

Without this feature, the new GUI was not usable in most non-trivial games. It’s hard to over-estimate the importance of this for declaring the new GUI production-ready.

Allow sprites to be dragged into scene view in 3d mode

Small but sweet. It’s a little thing, but makes a big difference to the frustration level of the user; in my experience, these things are often the real reason why you sacrifice the cost-savings and benefits of your own engine and use something like Unity instead.

i.e.: with your own engine, you can’t afford the time/money to polish every small interaction; 3rd-party game-engines can shine here, and should take more advantage of it.

I sometimes mention stuff like this as suggesting that Unity’s senior management team are out-of-touch with their product. Unity’s early community loved it largely because of small things, but as Unity grew, the company stopped sweating them. I suspect time will eventually tell that was a huge strategic mistake.

Improved Export Package window (now uses a proper tree view)

This window hasn’t correctly worked for at least three years; converting it to a “proper tree view” might – finally! – make the “Export Package” function do what it’s defined to, instead of exporting the wrong things, breaking keyboard and mouse interactions, etc.

(most of the bugs appeared to be a side-effect of the non-maintained, half-implemented, custom tree view code)

Make sure ObjectField and Object Selector shows icons for game objects and prefabs

Co-incidentally, I tweeted about this recently: Unity could make their engine noticeably faster to develop with if they provided visual feedback on these fields.

It’s embarassing, almost shameful, how poor the UX is on these fields today. By the sounds of things, this patch merely makes them “less crap”, rather than fixing them – but the last 5 years has been so awful that any improvement is welcome!

Selection.selectionChange callback triggered when selection changes

To call this “important” is somewhat of an understatement.

Substantial areas of the Editor API’s are surprisingly poorly maintained by Unity. The bugs undermine people each time they try to use one of Unity’s overwhelmingly greatest features: its extensibility.

Graphics: Added a quality setting ‘Shadow near plane offset’ to allow working around shadow pancaking artifacts

If this does what I suspect, it will make it trivial to fix a lot of common, frustrating “why do my shadows suck? OMG! UNITY SUXXS!!!!” situations.

Mecanim Improvements


I don’t use Mecanim: it’s been slow, clunky, buggy – and it solved the wrong problems at launch. It felt more like a marketing excuse rather than a developer feature. For me, working x-platform and often on mobile: not very useful.

However, looking at how long the list of fixes is … it might finally be worth using it. In context of all the other changes we’re seeing – maybe production-ready for general use now?

(modulo: the performance. Last time I checked, it was still way too slow for cross-platform games unless you were happy to have a relatively small number of animated objects)

Go check it out! If you’re a long-time Mecanim user, let us know what you think of this list of changes!

Now a TerrainData object embedded into a scene (created by script) allows scene objects being selected as tree prefabs

Unity’s Terrain system was implemented using a nasty, ugly hack that should never have been used.

As I understand it, that was in itself an attempt to workaround some problems with the fundamentally broken – and allegedly “unfixable” – Serialization system that is core to Unity, but is too much code for anyone to fix (and will be thrown out / fixed / replaced around Unity v6 or v7. Maybe).

This change reeks of a “workaround to a workaround”: you can’t create TerrainData in the scene (which should be the ONLY option!), but you can artifically create one if you write a script to do it for you. Either way, it’s good news – TerrainData Assets are a common source of project-corruption, because they’re abusing a feature that wasn’t intended for this.

Stacktrace logging: Allow to log full (native/managed) stack trace when log is printed, the option is available in the console window in top right corner popup menu.


I frequently discover major bugs in Unity core; there is never anything I can do about it because Unity strips / deletes all stack traces (in most cases) before outputting anything.

If this fix does what it says on the tin … WOOOHHHOOOO! … we can finally workaround Unity crashes (by debugging them!) and give Unity much more detailed and accurate bug-reports (leading to greatly increased chances of them fixing bugs).

Cross those fingers…

Fix Cursor

As I recently tweeted, Unity’s support for custom mouse-cursors was recently broken. That’s a huge pain given most games today use custom cursors. Several fixes here, looks like they’ve cleaned them all up (I hope).

Fix possible data loss on Windows when renaming folder with locked files


Fix occasional recycle bin bypassing when deleting assets on Windows


Serialization: Fixed not being able to serialize a field called “Base”


UI: Stop disabled graphics from blocking raycasts

If this is what I think it is, it’s another example of “should have been fixed before 5.0 was released” – but I’ve had so many problems with the bad system of Unity raycasts that I go out of my way to avoid them anyway.

Core: Fixed hang when passing large arrays to the Undo system

Sounds unexciting, but … I think I’ve encountered this on some of my projects: unexplained hangs that came out of nowhere and destroyed my ability to keep working on one particular game, related to Undo.

Note: I keep saying this in public: Unity’s Undo system simply Does Not Work, it’s appallingly buggy. As far as I can tell, the person who originally wrote it had little or no experience of what they were doing, and didn’t consult the literature to find out how to implement Undo systems correctly.

Some days, I genuinely feel it would be better for everyone if Unity removed the “ctrl-z” keyboard combo, because so often when you press those letters you corrupt your game project. A lot of the time, you don’t notice that Unity has just corrupted your project – not until much later, and then you don’t realise that it was Unity’s broken Undo that caused the corruption. Instead, you blame yourself, or your co-workers. Seriously, it’s that bad.

(I’ve stopped bothering logging bugs against it)

Importing: Changes to scripts are now correctly processed synchronously, fixing a range of issues with scripts relating to asset processing at import time

Subtle, but … I suspect this fixes a LOT of bugs we see on larger Unity projects, where you’re frequently hot-loading complex sets of scripts, assets, inter-related dependencies, custom asset-processors, etc.

…the end

So that’s it. What do you think? Any other major items you notice in the release notes that will make your life significantly better?

What’s still broken?

#Unity3d hardware usage + implications – Summer 2015

There’s tonnes of blogs out there, so I only talk about the bits that other people have missed, or were too polite or inexperienced to cover. Often that means I’m the one pointing out the flaws (most people don’t want to write bad things. Screw that; ignoring the bad points does you no favours!).

Sometimes I get to talk about the good bits that – sadly – few people have noticed. Here’s one of those.
Continue reading

New #unity3d feature: Virtual Scenes

Unity3D has a great core architecture – it’s easy to understand and use. However, it has some significant flaws.

One of the recurring problems I run into is that Unity requirs you to have precisely one “Scene” at any one time, but often you need to have multiple scenes at the same time. Problem? Yes, lots…

Unity 5 has some hacks to make it “sort-of possible” to have multiple scenes at once in-Editor.

Note: these were mentioned by Unity corp in August 2014, but aren’t available to most users yet

But they (appear to be) hacks: they help for a fraction of the use-cases. That’s fine: Unity has 10 years of software written on the assumption there is only one “Scene”; that’s an enormous amount of code to change! But can we do something about this?

TL;DR: I’ll put this on the Asset Store

If you just want a working solution, here’s the Unity Asset store page / support page, new versions will appear here first.

I’m using it myself on my own projects, and will do some beta testing with other people’s games. If it works well enough, it’ll go on Asset Store. I suggest reading this full post anyway – it’s an example of how to creatively solve small issues like this in Unity.

(if you’d like to beta-test this, tweet me @t_machine_org or email me (address at top of page). Unity Corp only let me have 12 beta-testers, so you’ll need to give me some info about what you’ll use it for, how soon, how much feedback you’ll give me, etc! Sorry :( )

Continue reading

Writing a #unity3d mini-game … to help you design a AAA game

One of my hobby projects is a testament to the Ultima and Elder Scrolls games – a massive open-world, where everything you do impacts the world around you. This has long been promised by commercial games – and it sucks from a gameplay perspective; it’s just not fun. But a very old ASCII-graphics game, Omega, showed a couple of ways of doing it that delivered on that promise while still being “fun” – very fun! That’s what I’m working towards.

It’s a test-bed for the high-performance Entity System I’m writing for Unity (more info here). I’m aiming for “gameplay considerably richer than Skyrim” with “graphics about the level of Oblivion”. To be clear: Oblivion was released almost 10 years ago! To reproduce those graphics today is fairly easy.

Challenge: Believable Cities that evolve over time

One of the things I have yet to see in any FPS RPG is believable cities, and more: cities that actually evolve. The nearest I’ve seen is the beautiful “large towns” of the Assassin’s Creed series (they represent cities, but the distances are way too small).

That doesn’t cover the “evolve” part though – even the AC cities are 100% static and unchanging. Hand-crafted, and because they cannot change (technical design choice) the player is incapable of altering them. You can’t burn down blocks, or build a new bridge. You can’t buy two houses and knock them together.

Hey, RPG industry, you can do a lot better!

This is one of the main new gameplay elements in my design. If all I managed was to take an off-the-shelf-RPG mod and add living, growing cities … I’d probably be content. What does this mean, though?
Continue reading

#unity3d missing docs: EditorWindow lifecycle

If you do any non-trivial customization of Unity Editor, you’ll almost certainly extend EditorWindow. But Unity (to date) refuses to tell anyone how to do this legally, or what the contracts you’re being held to are. Without that, many plugins on the Unity Asset Store are wrong and buggy, and sooner or later accidentally cause problems for users.

Their fault? Partly, but I don’t blame them too much – rather: Unity’s fault for failing to document (and failing to test assets before approving them!).

For those of you who aren’t happy shipping crappy, broken, buggy plugins (And charging for them), here’s a guide / reference for the lifecycle of EditorWindow. Extra info, explanations, and gotchas welcome!


I’m working on my own Entity System for Unity; want to follow it?

…Sign up here to be kept up-to-date on the project’s upcoming Kickstarter / Patreon / whatever-it-ends-up-being (if you’re already signed up, it won’t add you again)

#UX conveying critical info to public via number rating: Food Hygiene Stickers

How do you make good UX / GUI to present information to a mass-market audience with widely varying levels of education and attention-span?

This is a problem faced all the time by game designers and developers. It’s one of the most under-appreciated skills of the profession: good game developers know more about “presenting information and choices” than almost anyone else, anywhere. Because games are generally overloaded with info, and the dev teams have to filter that down and present it clearly – but if they get it wrong, the player’s in-game character dies, the game is lost, and the game itself tanks commercially. The developers then – ultimately – lose their jobs when the studio goes bust.

I’m going to start looking at non-tech examples of design and UX through the lens of game-design, and see where it goes…
Continue reading

Use shift-N to create new scripts in Unity Project Window, fast

In Unity, you create new scripts many times per day, and folders but there’s no keyboard shortcuts. I fixed this, and made the popup rather more intelligent than Unity’s built-in one. I’ve decided to make the basic version available for free here:

UPDATE: there’s a bug in the 1.0 version – it WILL NOT WORK if you set your Project window to 2-column layout. This is due to BUGS IN UNITY (not my code!). I have a workaround that’s in the Asset Store version, but is missing from the free version. I’ll patch it later

FreeIntelligentNew.dll – I’m not supporting this; it’s free: use at your own risk.

Or, for $2, you can get the latest version with full source code from Asset Store – I’m supporting + updating this.


Drag/drop the DLL into any folder named “Editor” in your project.

(required by Unity; this is how they make sure that Editor scripts aren’t accidentally included in your final game)


  1. Select the Project panel
  2. Hit shift-N
  3. By default, it intelligently guesses if you wanted a new Folder or a new Script. If it guessed wrong, hit “tab” to cycle through the options
  4. Choose a name and hit Enter (like normal in Unity). Done!


This works by peeking into some of Unity’s internals and detecting which EditorWindow has focus. If it’s the Project window, it uses some hardcoded constants (which differ between Unity versions, due to bugs they gradually fix, I think) to get the positions right, and offer you the popup.

I did it this way because Unity currently provides NO PUBLIC API for reproducing some of their core Editor features (e.g. where they temporarily change the render-style of the “newly-created item” in a Unity windowpane, so that you can edit the name in-line. Not possible for you or I!)

With a bit more hacking, I might be able to hook Unity’s own code and enable / trigger / alter the hidden features from keyboard, but … that’s on the wrong side of “Fragile; would probably break unexpectedly in a future version”.

How I’ve done it so far it sits independently of Unity’s code, and is very unlikely to stop working, or to cause bugs later on. YMMV…

(if you want to know more, get the Asset Store version, read the source. I’m happy to answer quetsions via the support links or in the Unity Forums page for this)

Entity ID’s: how big, using UUIDs or not, why, etc?

This has come up a few times, and I ended up replying on Twitter:

But that’s a crappy way to find things later, so I made a quick-and-dirty infographic with a few key points:


I’m working on my own Entity System; want to follow it?

…Sign up here to be kept up-to-date on the project’s upcoming Kickstarter / Patreon / whatever-it-ends-up-being (if you’re already signed up, it won’t add you again)

6 ways to massively improve the #unity3d AssetStore (for #gamedev’s)

Six months ago I tweeted a handful of obvious ways that you could make the Unity Asset Store greatly more profitable.

One of the Unity folk reached out to me, claimed that Unity was highly invested in improving this and asked for specific suggestions. So I wrote longer, detailed versions of each tweet and emailed them.

It’s been six months. No response. So … for Unity’s competitors, maybe looking to make/improve their own Asset Stores (or newcomers hoping to unseat the incumbents), here’s six obvious commercial improvements.

I’ve cut a few paragraphs I wrote to Unity about who I think their 3 main audiences are on the Asset Store; I included them as a “here are the assumptions I’m making” – I have no idea what their real audiences are. So I omitted that here.

NB: I’ve made the formatting webpage friendly, added some details, but this is essentially an info dump. I was too busy at the time to sugar-coat it – I figured that if Unity wanted to talk, I’d talk to them, and in person I’m really quite friendly and gentle. But at the time we were working 24/7 getting ready for a major exhibition, so this is a bit … terse.

Continue reading

Journalling for #entitySystems: #unity3d corrupted my scene; I’ve built a debugger for Unity!

Experimenting with shorter Entity Systems articles

This post is an experiment: rather than write massive, heavily-edited articles, I’m writing “shallow” versions first, which takes me only an hour or so. Then I’m saving the in-depth, detailed versions of each topic for future subscribers to my own Unity3D Entity System. This way I can cover a lot more topics in the same amount of time, and I suspect most people will get all they need without the deep dives I used to do.

Let me know what you think – @t_machine_org

The Unity bug that deleted my game prototype

A few days ago, I fired up my Unity3d test project for the new Entity System. All the data was serialized into Unity’s scene, everything was fine. I was expecting to see all my “registered” components (that are saved directly into the scene) exactly as they were last time:

Screen Shot 2015-05-26 at 11.23.54

Instead … I got this. Why, Unity – WHY?

Screen Shot 2015-06-07 at 21.29.31

On further inspection, Unity’s serialization system had stabbed me in the back: it combined data from two different situations (which is blatantly wrong and should be impossible – although since Unity has failed to document it … who knows what contracts it promises to uphold internally?). I’d like to know why…but the Unity Editor won’t tell me (they don’t let us see this data).

But I don’t really know what happened. Even with extensive logging, Unity lacks a solid debugger – and most of Unity’s code here runs BEFORE your debugger starts or AFTER your debugger stops. Any Unity bugs here are happening inside the invisible, opaque, known-to-be-buggy, Unity startup/shutdown routines. ARGH!

Software … has BUGS! OMGWTFBBQ!

Unity has bugs. Lots of bugs. So does all software. So what?!

Some (many?) of Unity’s many bugs sit in mission-critical parts of the engine. Rumour suggests that’s not because the devs are unaware of them – quite the opposite! – but because they’re too hard to fix. Any mistake in the fix, and it might cause more harm than the original bug (because these systems are so core).

This in itself suggests Unity’s internal dev teams maybe lacked a culture of TDD in the early days. My impression was that they do significant TDD now (yes? no?) but if it only arrived recently, then much of their code might be weakly tested or even untestable. i.e.: I try not to be too hard on them here, even when I’m suffering and there’s no help to be had.

In particular with Serialization: Unity doesn’t allow anyone to see what the bugs are. The Serialization system (along with some other super-critical-data-corrupting-bastard-evil-features of Unity – like the Undo system) has no API to let people outside of Unity corp see WTF it’s doing.

And we know it has bugs. But we can only guess at them, and how to workaround them. Unless you have a million dollars spare and can afford to purchase source-code access…?

Incidentally, this is why it is standard in the games industry to demand source-code access to your game-engine. This is non-negotiable. It has always been a black mark against Unity in professional / experienced gamedev teams that they are so secretive about their source code. It would be fine if they had a high-quality, Automatically Tested, well-documented accurate API, etc – but they don’t.

I have an idea … a new feature for my Entity System

At this point, I’ve already accidentally built quite a lot of the pieces of a debugger for fixing Unity itself. I’m recording the exact data, I’m storing it in memory in raw bytes (which sidesteps all the bugs in Unity’s undocumented serialization layers), and I’m displaying live data-visualizations in human-readable pretty colours, etc.

Enter: the Entity Systems Journal.

What’s a Journal?

If you don’t know, and you care about programming that involves data or storage, read this:

In the event of a system crash or power failure, such file systems can be brought back online quicker with lower likelihood of becoming corrupted.

What’s an Entity System Journal?

As many people have noticed, while experimenting with Entity Systems for their games, an ES is very easy to integrate with Event-driven programming.

You’re changing raw data which has no side-effects; that makes it very easy to write generic code that automatically converts every “change” into an “Event object” that the rest of your app / game can programmatically inspect.

With an ES, we’re going to use a Journal to record exactly what happened to your entire game-state, at all times, and in all places.

Journals in AAA Gamedev

To be clear: in an abstract sense this is nothing new, games have been doing “journalling” (But not calling it that!) since the 1990’s, when it was discovered to be a very effective way of making multiplayer / internet games run reliably while sending minimal data over slow network connections. Read about the early RTS implementations (Anything from Patt Wyatt’s blog, or the old Ensemble Studios gamasutra piece about 10,000 archers IIRC).

What makes this a little different is that we’re going to be a lot more … explicit … about it. Our Entity Journal will come with its own cross-platform API that’s easy to write tools for.


a.k.a. “lessons learned in software design and architecture over many years and from standing on the shoulders of giants”

  1. The API itself might be best expressed using one of the API’s we’re already using to access and/or manage data elsewhere
    • aka: the “don’t screw-up like XML version 1 screwed up” rule; XML (a data langage) invented a second language to describe its data. Until they woke up one day and said WAIT A MINUTE … WE ALREADY HAVE A DATA LANGUAGE!
  2. Whatever first-class objects exist in the API need to be few, simple, and generic
    • Journals are a bit like Entity Systems: the simpler the API, the more robust and easier to optimize performance – and re-use with all the world’s generic data
  3. The Journal is a lot like Quake3’s networking compression; however we build it, it OUGHT to be compatible with that when finished
    • i.e. once this is implemented, it ought to be re-usable “in server mode” with almost no code changes to provide a multi-host non-uniform distributed event log and playback system. “ought to”.
  4. From a tools perspective, we want to click on any Journal event and see the exact state of the entire Game at that point in time
    • Game Debugging … on steroids. BECAUSE WE CAN, THAT’S WHY
  5. …which’ll destroy C#’s crappy Garbage Collector.
    • So we’d better be aggressive about using weak references and/or converting “old” data into “summary” data, and dumping the data itself
  6. Just like an FS Journal, the ES Journal should allow us to reconstruct our entire game data at any point in time, so that we never, ever lose data, no matter how many bugs in Unity we encounter
    • Unity: your refusal to debug and TEST your own frickin’ “undo” implementation? Yeah. I’m looking at YOU!
    • Make a stupid mistake in your game-code and accidentally delete critical stuff while in Editor? Not done a git-commit recently? NO WORRIES! GO BACK IN TIME AND RELOAD IT! YEAH!
    • Find a bug in the Entity System itself? (shock! Horror! But also: guaranteed to happen) … well, we’ve got you covered ;)
    • …(which leads to the next point)
  7. This absolutely unfailingly guaranteeably … must not have any bugs in it.
    • Unit Tests FTW
    • This is our equivalent to Unity’s Serialization: everything will rely upon the assumption that “the Journal will take care of it”, making the Journal our weakest link / most critical code
    • KISS: the simpler we keep the implementation AND the API, the more confident we can be that it works at all times in all situations

Building it

Yeah, well … that’s going to be a much longer article (or three!)

Want more? Want to support this ES in future?

…Sign up here to be kept up-to-date on the project’s upcoming Kickstarter / Patreon / whatever-it-ends-up-beign (if you’re already signed up, it won’t add you again)

2015 talk: Progress on An Entity System for #Unity3d

Slides from my talk last night at Unity Brighton.

Three parts:

  1. First: introduction / overview of Entity Systems, and why you care (as a “person who makes games”)
  2. Second: Progress so far: screenshots of the Unity Editor with explanation of the new features I’ve added
  3. Third: Challenges and solutions I’ve encountered so far


WordPress, HTML, the web etc – don’t support “presentations” yet. No, I’m not creating a fake Slideshare account just so you can wait ages for a non-bookmarkable slide viewer to (slowly) load. Until then … we have PDF. Enjoy!

Self deception of the Startup Founder: Paralysed decisions

This week I’m paralysed on some of my simplest decisions while happily making complex decisions quickly, and being incisive and highly effective on others. This problem occasionally crops up in my work and personal life, and it frustrates the hell out of me; I’m going to write about it here, see if it helps me find a new way forwards. Hopefully this may also help others who’ve experienced a similar problem in their own lives.

Continue reading

Data Structures for Entity Systems: Multi-threading and Networking

(One of a series of posts about converting Unity3D into a 1st-class Entity Component System. Read others here)

It’s a year and a half since I wrote about choosing Data Structures for high-performance Entity Systems in A-AAA games. I always had a followup post in mind, but never wrote it down (thanks largely to the demise of #AltDevBlog). These days I’m too busy to write such big articles, but it’s long overdue … This post is a shorter version of what I had planned, but should have enough to get you going in the right directions.

Continue reading

Replacing Unity3d’s core architecture: Structs

(One of a series of posts about converting Unity3D into a 1st-class Entity Component System. Read others here)

How to store game data: GameObject? MonoBehaviour? ScriptableObject?

Those three pillars of Unity’s architecture are the curse of game implementation and performance. Architecturally, they’re great: they’re a big part of what makes Unity easy for newcomers to pick up and understand within hours, instead of taking weeks.

On large Unity game projects, they’re hated for being slow and hard to work with when you get to hundreds of thousands (or millions, tens of millions, …) in a single scene. Unity provides zero tools and support for this situation – it just gets harder and harder to work with, and slower and slower to run.

In Entity Systems world, we hate them because they are inherently slow and buggy – no matter how great Unity’s coders, they’ll never make them work as well as our preferred approach (See below).

But … replacing them forces you to replace most of the Unity Editor!

Most of the features of our Entity System need new Editor GUI anyway. If we’re going to be adding big chunks to the editor, replacing GO/MB/SO is going to be much cheaper than normal. And it potentially has huge benefits that are independent of the Entity System.

If we’re going to replace them, what will we replace them with? Custom versions that have almost the same name and work the same way? Or something very different?

What is Unity’s ‘GameObject’ anyway?

As far as I can tell, the real meaning of GameObject is:

def. GameObject: Something that exists inside a Scene, and has a Position, Rotation, and Scale. It can be “embedded” inside any other GameObject, meaning that any change to the other object’s Position, Rotation, or Scale gets merged onto this one. It’s the ONLY thing that can have Unity “Components” (MonoBehaviours) attached to it. Anything that is not a GameObject, and is not attached to a GO, gets deleted every time you load/reload/play/resume a Scene.

From a code perspective, looking at the backend systems, there’s a lot more to it. The GameObject (GO) has a slightly weird name, when you think about it. You use it so much during Unity dev that you stop noticing it. “Game … Object” … wha?

Other purposes GameObject serves

Here’s a few obvious ones (I’ll edit later if I think of more):

  1. Guarantees everything has the GetInstanceID() method
    1. Needed for Unity’s Serialization system to work (because of how it’s been designed, not because serializing requires this – C# has built-ins that would have done the same job)
    2. Enables the Editor to “select” anything in any tab/window, and that thing is ALSO selected in all other tabs/windows (e.g. click in Scene, and the Inspector updates to show the components on that thing)
    3. …probably loads of other things. It’s so commonly used I hardly notice it
  2. When doing in-Editor things, Unity can often “ignore” classes that are “not part of the game, because they don’t extend GameObject”
    1. Reading between the lines of official docs, and simplifying vastly, this is what happens behind the scenes. Especially with Serialization, I suspect.
    2. This can be a huge pain. It’s hard to debug code when “GameObject subclasses” magically behave differently to “all other classes” with no indication of why
  3. Since every GO has a Transform … you can do automatic, general-purpose, high-quality, Occlusion Culling
    1. NB: for this to work perfectly, you’d also want every GameObject to have a .bounds variable. Why did Unity’s original architects leave that out? Dunno, sorry.

How to store game-data: the Entity Systems ideal way

We know this, it’s been discussed to death: if your programming language supports it, use structs.

By definition, structs are the most memory-efficient, CPU-efficient, multi-thread-safe, typesafe way of storing data, bar none. There is no better way to do this in mainstream languages. C# and C++ both support structs; I think we have a winner!

There are probably some awesomely clever advanced Math ways of doing things better, or by using raw assembler and code optimized for each individual CPU on the market. Or, or … but: in terms of what’s ‘normal’, and commonly used, this is the best you’re going to get

But how will this work in practice? What about all the “other” roles of GO etc in Unity?

Structs in … Identity (GetInstanceID)

Simple ECS implementations often hand-around raw pointers to their Components; that gets messy with Identity.

But most (all?) advanced implementations have some kind of indirection – a smart pointer, or an integer “index” that the Component Storage / Manager maps internally to the actual point in memory where that lives.

If we turn that “advanced” feature into a requirement, we should be fine.

Structs in … Serialization

  1. Start writing your own version of the Unity Serialization system that only uses structs
  2. Finish it the same day
  3. Discover it runs 10 times faster than Unity’s built-in one
  4. …with fewer bugs
  5. PROFIT!

Structs are the easiest thing you could possibly try to serialize and deserialize. Unity coders probably wish that they could restrict us all to structs – it would make their live much easier in some ways.

User-written structs: special rules?

Do we need to add special “rules” to user-written structs that appear in the game?

Nope! It turns out that C#’s built-in Reflection system is able to inspect every struct, shows us every field/property/etc, and lets us easily read and write to them. Both private and public.

CAVEAT: structs break C# .SetValue()

Yeah, this sucks – and it’s nothing to do with Unity, it’s core C#.

A little care is needed when we implement the Entity System internals. But it has no effect on user code / game code.

Structs in … SceneView (Transforms)

No! Just … NO!

This is one of the architectural mistakes of Unity we’re going to fix in passing: we won’t require everything to have a Transform.

However, it foreshadows a more subtle problem that we’ll eventually have to deal with:

Entity Systems are a table-based/Relational/RDBMS way to store and access data; such systems are very inefficient at storing and retrieving tree-structured data (e.g. OOP classes + objects)

The transform hierarchy in a Unity scene is a massive tree. As it turns out, it’s bigger than it needs to be (because of that law of Unity that everything must have a Transform) – but we uses trees all the damn time in game-development. So it’s a problem we’ll have to come back to.

Bigger storage: where do the structs live?

Fine; at the lowest level, the individual ECS Components, we’ll use structs.

But how do we store those?

Choosing the right aggregate data-structures for your ECS is a major topic in itself. C# gives us (almost) total control of how we’d like to do it – plenty of rope to hang ourselves with.

At this point: I’m not sure.

As a temporary solution, I’m going to focus on making the storage-system swappable, so that I can implement a crappy version quickly, and then easily swap-it-out with something much better – without having to rewrite anything else.

Does it work?

Yes. I tried it – I wrote a complete replacement for the Unity Inspector tab that only works on Entities-with-Components, where “Component” is any struct. Ugly, but it works:

Screen Shot 2015-04-29 at 14.41.23

Everything there is auto-generated (including the colour; every struct gets a globally unique background colour, which I find makes it much easier when you have hundreds or thousands of Component types).

The only tricky bit was the problem with C# FieldInfo.SetValue(…), as mentioned above.

Further Reading

…some things I might want to pick up on in future posts:

Kickstarting Entity Systems in Unity3d – What to include?

A first-class Entity System for Unity3D

Unity is a great 3D engine and a good editor – but the programming environment is 10-15 years behind the curve. Entity Systems are a much better way of writing games, but they are surprisingly difficult to add to Unity.

I’ll bring the latest in modern ES techniques to Unity, turning parts of it from a “weak” programming environment into “one of the best”. In game-engine terms, this will put it far ahead of the pack.

This blog post is testing the waters:

  1. You can give me direct feedback on the proposed contents/scope
  2. I can start to see how much interest there is in making this

First round of feedback

A lot of people read this and got the impression I was going to charge them extra for the final product, and provide a library with no tutorials. Big confusion and misunderstanding! So, I’ve reworded those bits to be clearer.

Kickstarter: planning the campaign

Target Audiences (Expected)

I think I have four target audiences:

  1. Newbie devs, and non-programmers (artists, designers), who find the “coding” parts of Unity overwhelmingly difficult today
  2. Unity devs who’ve never used an Entity System on a real game project, and don’t know what they’re missing. They may have a lot of experience (shipped multiple games), or none (University/College students).
  3. Small Indie teams / sole coders who’ve used Entity Systems elsewhere, and feel that coding in Unity is slow, error-prone, boring, frustrating, and time-wasting by comparison
  4. Large Indie teams whose games won’t work in Unity because Unity is so bad at scaling to large projects. They’ve already re-written or replaced substantial chunks of Unity’s core so that it’ll work fast enough / reliably enough.

Types of Reward

Reward type 1: Discounted License

Everyone who backs the KS gets a copy of the library / license to use it. By backing the KS (at any level) you’re getting a substantially cheaper purchase compared to the final launch pricing on the Asset Store.

There might be some options here for different team sizes, different levels of support, etc.

Reward type 2: Expert Knowledge

When you have non-trivial questions about ES design and usage, it’s hard to get the attention of experts. They’re usually experienced, older, programmers with young families, or working in senior jobs. They have little free time.

Big developers fix this by hiring experts for a few days at a time. But small teams/individuals can’t afford the “entry” cost – as a Consultant, you need to charge at least $5,000 per engagement to cover all the admin overhead and hidden costs.

So, we have some options here to give everyone this kind of access.

Reward type 3: Source Code

In mainstream IT, when you purchase a library you rarely (if ever) get the source code. If you find bugs, or missing features, the vendor works hard and fast to fix them for you.

In the games industry, it’s the other way around: vendors take no responsibility for the code they’ve sold you, and in return … they give you full source access. “Fix it yourself, dude!”

If I let everyone have source, it raises my support costs. With each change, it won’t work for some people – because they’ve modified their copy – and I’ll have to help them update it.

On the flip side: Source code is your “Plan B” in case I get sick, or sell-out, or lose interest in the project, etc. It also gives you an easy way to add features of your own and send them back to me (which I then take on responsibility for maintaining).

I’m offering this, but it’ll be a high-cost. The teams that really need it should have no problem justifying the cost.

Types of Stretch Goal

To clear up confusion: these are things that would be present in the core product (e.g. the GUI is going to be a major part of the library!), but stretch goals would allow me to add extras to each of them.

Goal type 1: Editor GUI

In my experience the biggest effect on programming speed – once you get past your innate ability + level of experience – is the quality and “friction” of your tools.

As soon as you go beyond the simple stuff it’s damn hard to extend the Unity editor. But I’ve been hacking the Editor for a few years now, and I’m willing to do pretty much anything. Especially if it makes it easier to make games with!

Goal type 2: Example Games

The core will have some form of tutorials, but it probably won’t have a complete demo game.

Writing a game just to demonstrate an API is extremely time-consuming, and it’s not what you’re paying for when you buy the library.

But … it’s also a great idea: it improves the quality of the API, by giving me a reference product to check everything is as easy-to-use as intended. So I’d like to write one (or several), but I can’t do that on the base funding.

(Aside: If I ran the technology division at Unity corp, they’d be writing and publishing games every quarter)

Goal type 3: Tutorials

There will be tutorials with the core library. They’ll cover the basics of how to use it.

But … I’d like to go further, and do tutorials for every aspect. I’d like to do a written tutorial for everything, and I’d like to ALSO do a video tutorial for everything (some people only use written tutorials, other people only use videos. I’d like to make it great for both groups).

With a new library, tutorials are extremely expensive to write. Minor changes to the API – bug fixes, refactorings, feature additions – invalidate whole pages of tutorial at one stroke.

Pricing and Funding levels

Tier costs

These are approximate based on extrapolations of cost and expected “version 1.0″ feature-set.

  • License: $50-$150 (after launch, general public can buy at $200-$400 in Asset Store)
  • Source code: $300-$1000 (various amounts of support)
  • Access to experts: $150-$750 (various levels of privacy + interaction)

UPDATE: One person has stated they would “never pay these prices”. That’s fine – they’re way outside my target audience. I’m more likely to increase these prices than decrease them. If you know what an ECS is, and understand the value, this is probably too cheap already.

Funding targets

This is what it’s going to cost to build:

  1. Minimal version that’s usable in most game-dev: approx. $15,000
  2. My preferred v1.0: approx. $75,000
  3. High-quality version for large games: approx. $200,000

I can write-off about 50% of the cost as this is a project that I need and will use myself – and it’s a labour of love. I can also add a %age in expected additional sales that happen naturally after the campaign has ended (from the Asset Store, etc).

But Kickstarter takes a cut, and the government takes a cut, and Amazon takes a cut. In the end, I need a KS campaign to raise something like:

  1. Minimal funding: $5,000
  2. Preferred funding: $30,000
  3. High-quality funding: $75,000

My situation

Who are you, anyway?

In case you don’t know … I’ve run Technology/Development for a couple of game companies (e.g. Mindcandy, NCsoft Europe). My StackOverflow score is somewhere north of 20,000. I’ve contributed to a few open-source projects, but the one I’m most active on is the SVG rendering library for iOS and Mac, where I’m the project lead and one of the top 3 contributors.

I started writing about Entity Systems back in 2007, mostly because I was frustrated that so many game studios were “doing it wrong”.

What I’m doing now

I teach non-technical people how to program. I work with schools, teachers, and directly with children/adults. I invented a new system of programming using physical objects and toys.

Working with a local school, I’m making a new course where I’ll be teaching children “how to make games with Unity3D”. I’m particularly interested in what I can do to the Unity Editor to make it more user-friendly and better for use in the classroom.

Depending on how this goes, I have a few people in mind I’ll hire to share the burden of writing this library. It’s something I need both in my day-job and in all my personal game projects. Doing it as a Kickstarter gives me the excuse to spend 100x as much time and effort on it as I would otherwise, and gives me a much better library than I’d get on my own.

Why Kickstarter?

I suck at Marketing. The biggest risk factor to this project is that I fail to make enough noise, and hence there’s too little money coming in to pay for the ongoing development and support. As a result, dev gets sidelined while I pay the bills.

A Kickstarter is my way of testing “can I find a critical mass of people who need this project, and get them to put their money where their mouth is?”.

It’s going to be hard – I suck at marketing – but to be honest I’m leaning heavily on the idea that this is something people want badly enough they’ll help me market and promote it. With KS, there’s no risk to you for helping here … if the campaign fails, you pay nothing. If it succeeds, I get enough cash (and enough users!) to guarantee development through to a production-ready version.

Are you in?

Sign up here to be kept up to date on the project Kickstarter (if you’re already signed up, it won’t add you again)

ECS for Unity: design notes + next prototype

Recently I’ve been thinking a lot about how a high-quality ECS would appear / present itself within Unity, how it would interact with standard Unity3d game-code at the API level, how it would appear in the Unity Editor, etc. This evening, I had a go at making some of that within Unity itself.

NB: a few weeks ago I made a small Editor Plugin that used Reflection to upgrade and fix some of the core Unity Editor code/GUI. That was practice for what I’m doing here. It’ll hopefully appear on the Asset Store at some point (pending review right now!) – it has the full source unobfuscated, so you can see for yourself both what and how I got it to do its clever bits.

Continue reading

Intelligent “new (thing)” control for #unity3d

In Unity, hundreds of times a day you do “new (folder)” or “new (C# script)” etc. But Unity makes this ridiculously hard: you have to hunt and peck a tiny button and then hunt-peck a tiny item in a huge dropdown that is very easy to miss.

And you can’t do any of this from the keyboard. Even though you’re fully in keyboard mode and about to write the name of the New Thing (required!) and if it’s a script: type in the source code.

Screw that.

UPDATE: Now available in Asset Store – get the source

For a mere $2, I will Intelligent New you long time:

Source is included. If you want to see how I did it, knock yourselves out…

My context-sensitive New for Unity

So, what I’ve made today:


  • Select any folder or object in the Project window
  • Hit “shift-N” (because Unity sucks and steals Ctrl-N and refuses to let you choose a better one. Stupid stupid stupid.)
  • A popup appears pre-filled with a sensible new file name OR folder name
    1. If you were on a folder, it prepares to auto-create a new folder
    2. If you were on a script file, it prepares to auto-create a new script
    3. The name is pre-selected and editable, exactly as with Unity built-in
  • Sometimes that intelligent guess above will be wrong, so … hit Tab, and it switches type, while keeping the text selected and editable

Net result: super-fast workflow for creating new scripts and organizing your project.

(This required an ungodly amount of hacking and trial and error to work around bugs, bad documentation, and obvious missing core features from Unity APIs. But it works, and seems to be pretty seamless now)

Implementation Notes

A subset of the things I discovered / re-discovered on the journey to this one small fix / improvement:

  1. AssetDatabase.CreateAsset is basically broken: can’t create C# scripts at all. Don’t try.
  2. ScriptableWizard is unusable in Unity less than 5.0, because they made a core method private / non-overridable. Don’t bother.
  3. You can find the Project window and others by doing a reflection on the Assembly to find the magic C# Type of known Unity private classes (that SHOULD BE public!) and comparing at runtime. Hoops? Jumping through? Because an API has something private that should have always been public? Welcome to Unity customization! :)
  4. You can find the exact position of the selected row by adding yourself as a callback on Unity’s own row-by-row rendering of their built-in windows, and saving the data every frame. Sounds scary; works great (one of the few bits here that is a perfect hack with no downsides)
  5. Popups need you to create a new class, and that class MUST BE in file of its own or you will break Unity (internal bugs in Unity window management that trigger when you reload/restart Unity)
  6. …if you trigger that bug (or in any way end up with floating, non-closeable windows), go to the Layout menu and re-select the layout you’re already using. It will kill any unexpected windows. Phew!
  7. Unity still hasn’t made the Indent width (in pixels) public. The documentation insults you by saying that when you hardcode the number 20 this will break your code in future. YES, WE KNOW. SO MAKE THE DAMN NUMBER PUBLIC, YOU BASTARDS.
  8. There is a 2 pixel top and bottom padding needed to surround textfields in 1-line popups. I hardcoded this, it may have a number somewhere in the API. Good frickin’ luck trying to find it. Given the taunt about indents above, I very much doubt it is public.
  9. If you have a button or non-editable textfield with a changeable title, or anything that can be changed by a keypress during rendering, it is critical that in OnGUI you read Event.current.type and switch() on it. As far as I can tell, this has never been documented except in forum threads and blog posts by Unity staff. Once you know about it, you can sort of guess it from reading the undocumented bits of the API, but it’s damn hard to find references to it online. (I found it years ago by trial and error and blind luck). Until you do this you get race conditions when changing GUI based on keypresses. Ugh.


    Critically, this (should) work no matter how much I customized Unity elsewhere. It’s peeking into the actual window and render areas to decide what to popup and where on screen.

    The only conflict I expect is one that – thanks to some brainfart design at Unity corp – we can’t workaround: I had to hardcode “shift N” as the keyboard shortcut :(.

    Want it yourself?

    I thought it would take me about 10-20 minutes to do. HAHAHAHAHAHAAHAH! If I’d done this as a client project, I worked out it cost almost $1500 in billable hours. Ugh.

    I’m going to use this in production for a while. If it continues not to break or corrupt anything, I’ll put it on the Asset Store.

    Next up…

    …I’m going to do some context-sensitive New for the Hierarchy window, I think. Now that I have these hacks working, that’ll be quite a lot easier.