Upcoming Events! Community Event Calendar

Bugsmashers: Server Bugs Written Wednesday 15th of March 2017 at 12:00pm by StormyWinters

As per usual, anything said during the show is subject to change by CIG and may not always be accurate at the time of posting. Also any mistakes you see that I may have missed, please let me know so I can correct them. Enjoy the show!

TL;DR (Too Long; Didn't Read)

  • Today's bug is a strange one where if there are two or any number of players with a third player in a ship, and that player disconnects and comes back, he will spawn in Crusader normally, but the other players still see the third guy still in his ship and can still injure or kill him. So a player might be running around then randomly die because someone killed his clone.
  • What's happening is there's a desync between the information the local player and what the server sees, and the other players in the server and that the server isn't telling the surrounding players that the local player left the server.
  • Mark goes through the code and finds that the server was being secretive about letting people know someone left the server while being in a cockpit and ensures that the server correctly lets everyone know that someone left. 

Full Transcript

Mark Abent: Hey everyone, welcome to Bugsmashers. I’m your host, Mark Abent, I’m a Senior Gameplay Programmer here at CIG,LA and I’m here to take you behind the scenes on today’s hottest bugs.

Hey everyone, we are here in Crusader and I have two clients. Client 3 or number 2 and then client 1, the reason why I said 2-3 because dedicated servers 1 and then 2 and 3. So, have my two clients and I’m going to have the third guy spawn a debug ship, normally I would run up the terminal and spawn a ship but because I’m lazy and I don’t feel like running, I’m going to spawn a Gladius. Woo, so I spawned a Gladius and he’ll automatically get teleported into there, love debug commands and I have the other client, let’s see. We got to find where that ship is, let’s go to lightspeed, there he is.

So, we have a guy in the seat, pretty standard stuff and what we’re now going to do is go back over to the guy in the seat… guy’s running around, ruuunnnn. We’re going to disconnect, goodbye. Now we’re going to remove this breakpoint, there let’s disconnect and of course I will trigger breakpoints everywhere else cause of other debugging. So, as this guy processes his exit, let me explain the bug and why we’re here. The bug is basically when you have two people or any number and one guy is in a ship and he leaves when he comes back on his machine, he’s up out of the bed, spawning into the normal Crusader area… everyone else’s machine including some guy watching it like this guy right here. He will spawn back into the ship and unfortunately everyone thinks he’s in the ship except for the server and that local player.

According to both of them he’s running somewhere else around Crusader doing whatever he wanted to do and now we have a desync between all of the remote players to see and what those two… the server and the local client sees. What that also means is you get some weird oddities that allow you to kill the player in the ship when he’s not even there, he could be running around and then he dies all of the sudden because it’s reporting the wrong spot.

So, once this guy has fully disconnected we’ll see exactly what’s going on. Actually takes a little bit to disconnect so what we can look at is, in here in the player code when he deleted or destroyed before that happens we have to do a bunch of clean up and this particular section of the code is when the player cleans up. And as you can see here, when we clean up we’re telling the vehicle, this is… we call it the seatos because of shenanigans. We basically tell the vehicle, we get all of his seats and say, ‘hey, delete evict everybody and particularly evict this guy cause he’s getting deleted’. The server is supposed to tell everyone else, ‘hey this guy is getting removed’.

So it comes into this bit of code on the server and it will run exit but if you notice it’s running this thing called ‘Exit Int’ and Exit Int is the actual implementation to remove him from the seat. This will play an animation or teleport him to an exit position or whatever it needs to do. It’s the implementation that all clients will run however it’s not the code path that makes the server tell everyone to run it. It’s like ‘hey, this is the stuff they would be running’ so it’s responsible for running the exit code, the actual implementation but not actually dispatching it to let everybody know that ‘hey I should exit’. Instead of running Exit Int or implementation, we should be running Exit...come on, there if I could spell, there we go. We should be running Exit. Exit will run the implementation, execute Exit will do an animation and then eventually do execute Exit but it will also send the state to all the remote machines and since the server is not going through this particular path, it’s just skipping it and directly removing him… all the remote machines aren’t knowing that this guy shouldn’t be in the seat anymore.

So he’s still processing his exit and we shall go to reset, so ideally what we want to do in here instead is change this to exit so that it runs the same sequence of events except it makes sure it dispatches it to all the remote clients. All right, so now this guy’s back so this guy is still looking at the ship with nobody in it and we’re going to reconnect this guy in to see the bug. As we see the bug, what we want to do to fix it is check if we’re the server, make sure we exit and we want to skip transitions because we don’t want to play an exit animation and we want to force it.

Now the other thing we could have done is instead of in here where we dispatched the event, we’re going to have all the clients do it so that way they could also exit instead of having the server tell them. Which is true but we also want to make sure the server is authoritative for everything especially because of persistence and we have the same entity IDs we need to make sure that everyone’s up to date and not just because some guy thinks a certain state should be in.

So, having the server dictate these things is always a bonus. Now there is code elsewhere that handles removing the player properly when he’s in the seat but it won’t correctly reset his entity ID and that’s why we need the server to tell everyone that this is the case. Especially since before we delete the entities, it’s up to the server to say which stuff was deleted.

So if the server says, ‘hey remove this guy from the seat’ followed by delete him then we insure all the clients have the same logic. Load the level… so this guy spawned in, he thinks he’s in the bed, the server thinks he’s in the bed however that other client thinks he’s attached and he even has broken animations. So from the perspective of the remote client cause it would be everyone and this guy that are out of sync, so let’s give it a try with our fix, let’s see what happens.

Compile, compile, compile… this is the part they don’t show you all the time. Yeah, we might fix a bug but it takes a few bit of time to compile everything and restart all the servers. They’re finally fixed, then we give it to QA, make sure they verify it, then hopefully a day or two later it gets in. Or a new bug arises because we just uncovered another thing this thing was hiding… ah, game design, dark development.

All right, we’re back in business so let’s… fast forward, spawn the Gladius all over. Now the other guy who’s magical cheats to fast forward, let’s see where did he spawn. Weee, all right so there he is, now we’re going to exit, bam. Now we’re processing the exit so the server will tell all the remote clients, ‘hey this guys has left’. As you can see, he left and the other guy is slowly leaving and I just realized that true actually enables transitions where we don’t want transitions on.

Just to be extra safe before committing, we will remove that and what this will do is allow the local clients to perform this as well. However the server will still dictate the exit so that the local client will make sure to use that state and just to reinforce it, the server will say, ‘hey, flush it’. With those two things the bug will be good to go, and once this guy… client two disconnects, we’ll reconnect him and we’ll see he’s no longer in the seat when he reconnects and he’s back in his little cubby hole on all machines. All right, as you guy see he connected fully and he’s no longer in the ship so the bug’s fixed, problem solved. Hope you guys enjoyed, til next time’s exploit.

So, as you guys saw we had a little bit of bug on the server where when a guy was leaving the server and he was inside of a seat, the server would remove him from the seat and wouldn’t tell anyone else about it. We have everything being server authoritative so if the server doesn’t tell the clients or remote clients that something happened, things get a little bit wonky and because that server didn’t tell everyone else that this guy left his seat… everyone else thought he was still there. Once he joins since we have persistent numbers for what our players are, our entity IDs, everyone thought he was still in the seat when he reconnected so all the other remote machines thought he was still in the previous seat and even though he was maybe miles away when he rejoined. From a perspective he’s still in the vehicle but he’s not, now the server correctly informs everyone he has left and everyone sees that he has successfully left the seat so when he reconnects he won’t be in there, then that can’t be used as an exploit. Hope you guys enjoyed, til next time.


Director of Fiction

Moonlighting as a writer in her spare time StormyWinters combines her passion for the written word and love of science fiction resulting in innumerable works of fiction. As the Director of Fiction, she works with a fantastic team of writers to bring you amazing stories that transport you to new places week after week.