As per usual, anything said during the show is subject to change by CIG and may not always be accurate at the time of posting. Also any mistakes you see that I may have missed, please let me know so I can correct them. Enjoy the show!
Mark Abent: Hey everyone, welcome to Bugsmashers. My name’s Mark Abent, I’m a Gameplay Programmer here at Cloud Imperium Games, Los Angeles and if you guys have never seen this before it’s pretty cool. I take you behind the scenes of some fancy bugs that disrupt, you know, your guys play time and even some design errors or even our stuff. You know it’s pretty cool to see what happens and why and I give you a quick breakdown of what’s happening and the fix. Well, hope you guys enjoy, let’s take a look.
Hey everyone, we’re here in my fancy, dancy test level and Bob here is annoying me so I’m going to get rid of him, bye Bob. All right, so we’re here to test out the a fun crash bug and certain entities get destroyed. If Bob goes away he may crash but let’s speed up this process, we’re going to spawn a fun Constellation...wooo, Constellation. I’m going to shoot the crap out of it and hopefully we get a crash that QA was reporting.
So apparently with this bug when you destroy and delete things we’re getting a crash with some corrupt memory so let’s see if we can replicate that. Everything’s dancing, weee, ahhh and crash, excellent. So what’s happening here? Well, when we try to remove these entities we get this thing called an ‘event done’ and this gets dispatched to all sorts of systems to say, ‘hey clean up, we’re about to delete this entity’ and unfortunately there’s some kind of memory corruption happening because in this particular area it’s supposed to stream out what.. I put some debug code in here to what’s actually getting deleted and unfortunately the information we’re getting out of this is just pure garbage.
What’s happening after some intense debugging, we have two competing systems racing for the ‘event done’ listening. We have this thing called an item port which is…if we have ‘keys’, I can attach the item or should I say these ‘keys’ to the ship and the ship could fly away and it has ‘keys’ or guns or whatever and we have this thing called a physics controller. So when these keys goes onto the item port, it will lose it’s physics but then physicalize on the ship. Basically the physics become one with the ship so when the ship gets hit, we know which item gets hit. It’s some clever magic and unfortunately when they both register for this ‘event done’, depending upon the order of which these components were created… the physics controller may get it first or the item port controller may get it first. If the item port controller gets it first, it will actually detach these ‘keys’ and say, ‘all right, you’re done’. The physics controller will then get informed that these ‘keys’ have been detached so it should basically now own it’s own physics but if the physics controller gets it first, it will think it’s detached and then it will physicalize itself.
So now, it has physics on the ship and physics on itself but unfortunately it’s only aware of one at a time so it gets rid of the one on the ship but the ship still thinks it has it, this is where the issue lies. So now this thing has it’s own physics and it’s still attached and when the item port gets the event it tells it to detach and physicalize but the ship still thinks it has the ownership of the physics so it now tries to clean up to rebuild it and it’s now pointing to an old point memory... it will touch and corrupt anything now that’s in that memory position. Then things just go haywire, you start crashing in random positions so the correct fix is actually to get rid of the event listener.
We don’t really need for the physics controller to listen for these events cause we have the item port system to do it for us. This was just an extra set of protection that we don’t really need, in fact it’s causing a lot of harm. So this is one of those simple cases where extra safety actually caused us harm since we don’t really need it. So the fix is actually just to gut out the system so it doesn’t listen for those and instead it listens… it gets told by the item port system. So, I’m going to gut all the locations where it registers itself, weee.
Then clean up the listener set, oh where’d it go… goodbye listeners and there it is, aha. There it is, so as you can see if physics got it first it would pretend it was released even though it was still attached. So it’s just a safety measure in case the item port system was not reliable but the item port system is reliable so we can just gut this old code path. All right, let’s rebuild through the magic of editing will be done now.
So, we’re back in the fun little level, Bob’s twitching out so let’s take care of his misery. Oh and made the ship go boom, excellent. Ah, game design. So, we’re now going to spawn that fun Constellation and see if it’ll crash. Where is it spawning, there you are, where’d it go, there you are… it moved. So, let’s shoot this thing and before you say, the pistol’s destroying every single thing I may have a developer gun that lets me destroy things, oh that looks awesome.
All right, when this thing goes away that should be the vehicle, it shouldn’t crash. Aye, it didn’t crash so thanks to my little debug we can see when that event got dispatched, the radar, some flow graph and physics… that’s Kythira, all got information that’s not physics controller. It’s just, yeah, labeled differently.
So everything cleaned up, heck we even got the item port stuff there so something got item ports so now when the item port gets the message it will detach the thing, the physics controller will learn about it and then it will send it off and everyone goes off their merry way, no more memory corruption, no more crashes, everyone’s having a good time. You can play without getting disconnected from a crash, hope you guys enjoyed.
So, we had a very exciting bug. You know when you’re in multiplayer and you shoot some random thing and when the entity goes away, boom, crash. Now your game’s gone and you have to restart all over to play again. Not too fun, so took a quick look at it and it was just an issue with the physics controller trying to do an extra step of protection. We have that in there originally because our component system wasn’t as robust when it was originally implemented, many, many, many months ago. Now we have a much more robust system so the flow and the expectations of when things would happen are more reliable. So didn’t need that extra step of protection now we don’t have that memory corruption, everything works as expected, no more crashes… everyone’s good to go. Hope you guys enjoyed and remember praise be the lamp.