As per usual, anything said during the show is subject to change by CIG and may not always be accurate at the time of posting. Also any mistakes you see that I may have missed, please let me know so I can correct them. Enjoy the show!
Review of physical shops, focus on missions and getting Item 2.0 ships set up from tech design standpoint and felt to be feature complete except one minor bug
Working on Item 2.0 HUD and combat indicators for those ships
Review of commodities felt to be feature complete and turning focus to stability and performance
Review of shop locations with directors, went well with a few bugs to be fixed but ready to release
Going through each commodity to set each profit margin range, so there isn’t just one lucrative tradeable commodity
Miles Eckhart in QA’s hands for testing
Ships out to Evocati for testing now
Headlights working now as well as ship bed logout functionality with persistence in that location
First pass of Item 2.0 items balance
Created a new light group so players can control their ships headlights/spotlights
Lead engineers have been working on performance related issues to improve the traversal experience
Working on the quantum travel VFX: switched from a state based system to a velocity driven system
Implemented dynamic nav points so inner nav points around a planet are created dynamically when it's targeted
Closed a lot of control and stability issues relating to grav lev vehicles and ensured they played nice with the atmospheric flight fixes
Fixed an issue where the streaming system stopped updating by switching from absolute server time to frame delta time
Working on displaying radar and scanning gameplay to provide information, control and feedback via the UI
Starmap is PTU-ready. 1 fix will go in after PTU is launched to be done before 3.0 Live.
Mission Manager has been PTU-ready for 2-3 weeks. A few more fixes will go in between PTU and 3.0 Live.
Successful evocati tests with 60 players in a server. Successful internal tests with 128 players in a server.
Most of the crashes and disconnects have been fixed. The remaining ones are rare and very difficult to reproduce - more players will help track these down.
A lot of optimization work has been done - for instance limiting which ships were being updated at any given time to ships in your vicinity or mission-critical ships. IFCS updates are large and taxing.
Ripped out net serialize from two of the last three systems that used this old CryEngine tech. Replaced with CIG’s serialized variables. Last system is being worked on and then net serialize can be discarded entirely.
Performance is much better recently with essentially all areas having at least a steady 30 FPS. Most areas higher, into the 40s and 50s.
179 issues last Friday, now on initial PTU push.
Sandi Gardiner (SG): Hello and welcome to another episode of Around the ‘Verse, our weekly look at Star Citizen’s ongoing development. I am Sandi Gardiner.
Chris Roberts (CR): And I’m Chris Roberts.
SG: Today is a holiday in the US so our teams in Los Angeles and Austin are enjoying some much-deserved time off.
CR: Yes, they definitely are.
SG: Because of this, we don’t have a feature for today’s episode - but we do have some exciting news.
CR: Yes we do, that’s right, as I’m sure you’ve heard - Alpha 3.0 has gone to a wider PTU, so we’ve gone beyond evocateh… avocoti…
CR: Evocati, as you would say…
CR: So, let’s see what went into getting the build ready for this expanded group of testers in this week’s burndown.
Eric Kieron Davis (EKD): Welcome back to Burndown, our weekly show dedicated to reviewing the process of the release of Star Citizen Alpha 3.0. In the past week we’ve made tremendous progress towards the release of PTU as we focused on the categories: shopping, missions, ships and vehicles, traversal, mobiGlas and overall performance and stability. So let’s check in with the team to see how each progressed.
Todd Papy (TP): This week we did a review of the physical shops which from my standpoint they were feature complete. Then we’ve been focusing on missions and getting the Item 2.0 ships completely set up from tech design standpoint we feel we’re feature complete on that except for one minor bug. Then from there we’re still working on the Item 2.0 HUD and combat indicators for those ships and getting that to feel right. Then we’re doing a review of commodities today and I believe that’s feature complete and that will get turned to green and from there we will continue to basically focus on stability and performance. One of the things we knocked out this week was a Raycast issue that was causing memory crashes and overall performance issues so ideally the Evocati seen a gain in performance based off of that but that’s been plaguing us for a little while so getting that thing knocked out was good. From there it’s just been general stability and performance issues we’ve been focusing on and then everything the team’s been working on has really been going towards those tier 0’s that we’ve called out.
Jake Ross (JR): So this past week we had a review of the shop locations in 3.0 with Chris, Todd Papy and all the other directors. We went through each shop, we experienced the shopping flow and experienced… got to see the shopkeepers in action and it went really, really well. There's a few bugs we’re still working through to get fixed but overall the shop locations are in really good shape and ready to release in 3.0. So we’re in bug fixing and polish mode now and there’s plenty of stuff still to do, lots of bugs to fix up but we’re ready to release these things live to you guys so we’re really, really excited about that
Pete Mackay (PM): Been going through commodity by commodity and setting the profit margin range. Each commodity has its own profit margin wage which scales roughly with it’s base value but it’s not a linear relationship. So that’s taken some time to go through and just making sure that there’s never any one commodity that is always the best thing to trade, cause that’s what we don’t want to see. We don’t want players to get to a point where there’s just one trade route that’s the most lucrative and that’s it, so a lot of my work the last day or two has been just making sure that we’re in a good spot there.
JR: So this past week we got Miles Eckhart, one of the mission givers on 3.0 into QA’s hands, so he’s being run through his paces. QA is kind of taking him through his flow, identifying issues and calling out blockers. Those tech blockers are being looked at by our developers and getting resolved at a good pace and we’re turning attention… we have turned attention to Ruto as well and getting him all situated. Pretty soon we’ll be able to get him into QA’s hands as well so the mission givers are coming along pretty nicely.
JR: On the ship set up side, we’ve finally got the ships to a point where we’ve gotten them out to the Evocati and we’re ready for them really to bang on them and test them in their fullest capacity. On the setup tasks this past week we’re were able to knock out the headlights, getting all the headlights set up so they actually work so when you’re going onto the dark side of the moon and you want to explore… headlights will actually turn on now, we’re toying and tweaking with distance and brightness but for the most part they’re all set up now. We also got the beds in all the ships set up with the saved game logout functionality so when you get into the bed, you can actually log out of the beds now and persist in that location. So we’re really excited about that functionality and that will be online in the live environment here pretty soon as well. We have a big spreadsheet full of the different ships and all the different set up features we need to get implemented and at the start of this endeavour they were all red, all red cells meaning they haven’t been done yet. For the most part we got them all the green now, so we got a nice pretty green spreadsheet, lots of things been accomplished this past week and we’re ready for lots of testing to be done on those ships.
Steven Turberfield (ST): So one of the things we’re looking to implement with 3.0 is persistence spawning. So one of the things that tech design are doing at the moment are going through all the beds ingame in the ships and adding interaction points to those so that the player when they get into the bed they can basically save their progress and that will log them out as sleeping in the bed basically. So what we’ve been doing at the moment is adding interaction points to every single one, setting them up in data and basically saying when the player interacts with this specific point we’re telling them this where they’re logging and out and that’s where they will remain until they log back in again.
Kirk Tome (KT): For the 3.0 ship Burndown, we’ve been making a lot of progress fixing bugs on ships particularly the Caterpillar, Glaive, Scythe, 300 series, the Mustangs, the Aurorae, and the Constellations including fixing those bugs and implementing new features such as the new bed interaction feature which allows you to save and exit interaction on them. We’ve also implemented the new light feature which allows us to have controllable lights which gives us the ability to turn on headlights in the ships so you can see to the dark areas of the planets. We’re currently in the midst of doing the first pass on Item 2.0 items balance so that we can do things like actually control the power plants and shield generators and the weapons and make meaningful choices when we’re flying the ships and sliding those dials on the entities we just fine tuned and getting the features that you can control on those entities working.
Max Hung (MH): So we have this ticket that says the ESP is broken. ESP is what helps you stay on your target when you’re flying across space and you want to dogfights and it helps you lock your pips so it’s easier to land your shots. So the tickets says it’s broken after quantum traveling so the ticket was first sent to Michael who works on quantum drive and basically we took a look at it and figured that it doesn’t really involve quantum drive at all. So I jumped in tried to figure out if it involved targeting so because targeting is what shows on the pips and ESP is broken if you’re target is inaccurate. So, me and Michael were looking at it for a couple hours and we jumped all over around the universe, we locked onto different targets both allies and enemies and we couldn’t figure out what’s going on. So we have a hard time even figuring out if it was working on not or if it was on or off and so we started looking at the code to see if you could turn ESP on or off and turns out you can. We followed that trail down, turns out that the UI for ESP is broken so it always showed that it was on so we figured… if we do this quick fix at least it’ll be accurate and show that it is on or if it’s off so I put the fix in and then we went back and flying around and locking onto different targets and tracking them we still can’t figure it out. We looked back to what the ticket supplied and it said that ESP is broken and had a 5 second video of the swinging back and forth against a target. We spent a couple more hours and we started feeling like if none of us can figure it out and QA and can’t figure it out then probably it was a user error because the different keyboards that are involved in just playing this game. It might be possible that he hit and actually turned off ESP, the UI didn’t update so you don’t know that it’s off. So when you try to lock on and it’s broken so we put that fix in and everything looks fine, we’re sending it back and we’ll see how QA feels about it.
Matthew Intrieri (MI): Many days I come in and there’s a fire to put and today was no different. There was six major issues blocking us from releasing to the Evocati and eventually the PU. One of those issues was all the ships were missing, their LODs and the LODs are the level of detail meshes that as you get away from the ship it goes lower resolution. What was happening was that it was showing a very low resolution model up close and then it would disappear as you went away from the camera. So we had a lot of missing assets and you would usually think this was an art issue but we know that these are working in the game not too long ago so the fact that all 30 or 50 ships suddenly were missing was very curious. So we started looking into the issue and we thought, well let’s test this on our development build, let’s test this in the editor and it was fine in the editor, fine in the development build. Then we went over to QA and asked them to look at it and they noticed that it was only happening in the shipping build, now shipping build and development build where we work is a little different. There’s little different ways they build those processes, different flags that get set, not as much information gets put into the shipping build because it’s a little more streamlined. So, we were looking at since we’d isolated it to only the shipping build we started looking at the data inside the shipping build and we can tell that the build process had not put the correct files into our P4K PAC files. So once we realized that we contacted our dev ops engineer, Miles and he was able to clear the cache and solve the problem.
Michal Kozlowksi (MK): This week we had quite an interesting set of tasks given to us by Design. We started from this one bug, which was related to a … basically to a headlights on a ship not working properly, which evolved into creating a new system of light grouping for headlights for every ship on 3.0 and in a future development where players are actually able to control and toggle headlights - or spotlights - on their ships whenever you are in a dark area or you want to illuminate the surface of, let’s say an asteroid, when you are looking for something. Which is quite an interesting and very, very strong visual. And it’s something that was missing from ships before. So because it had to be done on every 3.0 ship and every ship has it’s own separate task it also actually resolved quite a big chunk - about twenty I would say - tasks on our to do list before we are ready to push for PTU. And it was quite a … quite a quick task. It literally took me about one hour to do. All you had to do is separate the headlights - or the spotlights - from the existing lighting rig into its own layer, assign a new light group just for them, and just tick a box which allowed players to use a shortcut to turn them on and off. I had to do it for every single ship and just like that there were twenty tasks … were done in a very, very short amount of time.
Ashram Kain (AK): This week we’ve been focused on getting things ready for when we go to PTU. Part of the big piece of this is updating our traversal experience: how ships fly and travel through our expanded universe. A big piece of this has been updating our navigation points for quantum travel as well as doing a lot of fine tuning on atmospheric flight systems, as well as some of the more interesting flight systems we have for vehicles like the Nox and the Dragonfly. Those will be really exceeding … exciting to see in game so look forward to that.
But this has come with another challenge. And that is making sure we have the performance that our fans expect. So we’ve had our lead engineers Chad McKinney and Mark Abent working on performance related issues in order to bring that experience up to high fidelity and high speed that we hope to see in the finished product.
Michael Dillon (MD): For the quantum drive we’ve been working on a lot of visual effects work recently. So we’ve took the old system which was using a different set of triggers and such - it was basically state driven. Every time we entered a new state of the drive - the spool, the ramp, the travel - we would just “Okay, fire off these effects”. That took a lot of the … a lot of the control away from Design on how they wanted it to behave. So we’ve instead changed it to a velocity based system. So now instead of it being this effect plays when we enter this effect … this stage of the drive - so spool up play this effect - it’s now when the ship reaches this velocity play this effect. It gives designers a lot more control over how the visuals are going to look. And gives the artists more control over how things develop through the flow as well as controlling our short and long range travel.
We also implemented our dynamic nav points so the inner nav network around a planet no longer has to be placed by designers - it can be generated dynamically and controlled based on what you’re actively targeting so it doesn’t flood the player’s screen with all kinds of useless data. If they don’t want it they can drop their quantum target, they’ll go away, have combat and then bring it back up whenever you need to.
Those have been the big things we’ve been pushing for this past week or so for 3.A0.
David Colson (DC): So in the last week I’ve closed out a lot of issues with the help of John to do with grav lev being quite unstable. So things like, for example, the grav-lev bikes flipping upside down and getting stuck in the geometry and just not being able to right themselves. So now they can fix that problem without much issue.
We’ve also had things like strange forces that just come out of nowhere and kick grav lev bikes up into … space, I guess. So those have been resolved as well. And general issues where grav lev bikes lose control: we had one issue where you would turn left - specifically left - and the grav lev bike would tumble and then carry on. If you tried to go right it wouldn’t happen which is a bit strange. We fixed that issue as well.
We still haven’t quite got it yet but we’re also looking into issue where there are these huge forces that occur whenever you drive a grav lev bike upto a big rock and grav lev just panics and goes “Oh no you’re going to hit the rock” and it just flings you up into space. So that’s been sort of limited.
And I think with that issue and the atmospheric stuff that John’s been working on - where you get this turbulence that I mentioned earlier - just making sure that they all work together and there’s no other issues on top of that. And pretty much at that point we’re PTU ready with this - this is ready to go - and it feels a lot nicer than it used to.
Muhammad Ahmad (MA): Okay I was investigating a PTU bug where as players are flying around the PU new objects were not rendering in. It was just basically missing from all the locations that you visited. The issue was that the streaming system was essentially stuck so as new objects were requested to stream in they were not getting updated and therefore they were not getting loaded, and not getting rendered.
To resolve the issue we first had to gather the information … some information from the Evocati and QA where we basically exposed some debug streaming information into the release builds so that we could essentially see if … what kind of issue was happening with the streaming system. Was it stuck? Was it running out of memory? Was it over budget? That sort of thing. Once we got that information we realised that essentially the streaming system was stuck and as new objects came in they were not getting updated - it wasn’t going over budget - it just wasn’t getting updated.
And to resolve this issue we essentially found the cause was that the streaming system relied on a check to update every 0.1 milliseconds and as you were connected to the server it was storing that value in a float which after a while it lost precision and so it couldn’t represent 0.1 milliseconds any more and therefore it never go into the block to update the importance value of the objects. And it never got streamed in. And basically to resolve it we changed that timer change to work on a frame delta time. Rather than server absolute time it was just looking at the difference - how much time has passed - since the last frame. And that resolved the streaming system bug.
Calix Reneau (CxR): I am primarily working on hooking up the radar gameplay and the scanning gameplay to the UI and the MFDs and the visor interface that will let you know that you are radar and scanning gameplay...ing. That it happens at all. Because the underlying mechanic is that … is a location versus another location versus the ambient signal between versus what you’re doing and what they’re doing all of which is basically invisible. So we put it on a screen and said “it happened” or “it didn’t”.
And so part of it is getting readouts from the things you’re scanning and reporting that information on screen in a way that makes sense and is generic enough that we can use it for debris or a vehicle or a planet or a jump point or whatever else. To the point to where we’ll want to be able to extend this to be able to … police scanners to scan the ship and its cargo and its crew and have the feedback on that go over time and let everybody know that it’s even happening.
So that’s the … once you have that information I … part of it is displaying it, the other part the gameplay of getting it, of manipulating your scanner to focus in on one part of space or to expand out to get a more broad sweep of the area. So all the controls to manipulate that and demonstrate to you what it is you’re doing when you push buttons and gameplay happens is … that’s what we’re doing.
Hugo Silva (HS): We find multiple challenges with displaying information to the player. Right now all of the ships have their own kind of display systems and we’re adding a new one - basically for scanning and trying to locate other players. This is a challenge because we don’t want to clutter the players with too much information. We want to make it the essential for the player to - if they want to scan something - they can easily do it and they can get all of the information in a simple and clear way to make them … to make better decisions, react faster. Just basically make gameplay more fun for them.
Leo Vansteenkiste (LV): So last week we had a few more polish tasks left for the Starmap. These have been done, so the Starmap is basically PTU ready. There is one more task I need to submit. We are not submitting it right now to prevent any instability for the first PTU release. But, it will probably be submitted for the next patch after that.
On the Starmap to radar conversion, I’ve been able to get some more work done. We have now a nice transition between the Starmap and the radar so if you zoom in and zoom out it will all go smoothly. I’m going to get a video together today and show it to the team to get some feedback and see where we go from there.
TP: In regards to the raycast issue that the programming team took care of, it took roughly a day and a half of all of the senior engineering programmers here in Frankfurt to track that sucker down and kill it.
Mark White (MWT): So the mission manager, that’s been PTU-ready for about 2 or 3 weeks. This last week Todd again reviewed it with some of the other designers. They identified the last few fixes that they’d like, so right now we’re in the process of estimating those out, getting them into the schedule, and then we’ll see where that falls for getting to the last few fixes before we go to 3.0 Live.
Live Release Sync
CR: I think we’ve got everyone. We’re gonna start the live sync. So, you want to take it away Mr. Webster?
MW: Firstly, we pushed to evocati on Friday and again on Saturday which is pretty good. We got a feedback report from Will Leverett that QA responded to in the last hour or so, highlighting the bugs that Will called out in there for the ships. The server stability, I’m just trying to take a look at it now, for the most part it seemed not too bad, I couldn’t see anything in there that highlighted anything particularly dreadful. Of course, we can ask the guys in the meeting in a little bit. Other than that, the main thing is to sort of highlight where there’s a couple of bugs that we already know that we’ve got that are being taken a look at where if ships go into a green zone they sort of stutter around a little bit.
Erin Roberts (ER): I was looking at the bugs and we need to push to PTU tonight even with some of these issues because we need to get those server statistics for tomorrow…
Unknown Speaker (US): Evocati.
ER: To evocati, sorry, yes. So, we will go live to evocati. Let’s see if we can find and fix some of these issues before we do that and whatever happens we need to go because we need to get the statistics from the server so we have that information so if we do fix the other issues tomorrow, we know whether we’ll have a nice stable server for larger PTU.
JR: Performance-wise, we’ve been doing lots of tests at increasing player count. So, externally we got up to 60 players in an instance. So, that was really really cool to see. We got lots of cool images of Port Olisar and all of the ships flying around. It’s really really interesting to see the different… how even just increasing player cap like that can make it feel a lot more lived in, a lot more exciting. Internally, we even got up to 128 players, so we’re doing lots of tests to try to increase player cap and it’s going really well so far.
Chad McKinney (CM): So, we’re winding down with the 3.0 release right now and that means that some of the things that we’re looking at are stability and performance and this is something that we’re always concerned with but when we’re really approaching a release they become higher in priority and these are two kind of broad topics.
Stability is something that, like I said, we’re always addressing but we’ve been fixing the obvious crashes, we’ve been fixing the obvious bugs and disconnections and now we’re getting to a point where we’ve gotten through most of the major issues and we’re left with either very difficult to solve crashes or very difficult to reproduce. So, often times I’ll get some report from the evocati and it will be that one time during an evocati run there was a crash on the server for some reason or a disconnection and I get a callstack, I get a core dump, and there’s no repro - we’ve only seen it once - and we need to fix it somehow. This has come up many times. I have to look at these kinds of issues and just try to figure out how it was even possible that, say, the server can get into a particular state and how to fix it without ever seeing the crash. So, this requires a lot of collaboration, for instance with QA, to get certain kinds of information. Working with evocati so we can try a fix that may not hold and kind of go back and forth until we get something that looks like it’s working. This is an ongoing process and this is the kind of thing where having the evocati is incredibly useful because it means that we can test at a scale that we’re just not able to achieve internally. So, that’s one big set of things that we’re looking at right now.
Mark Abent (MA): One of the few things that we’ve been trying to do hard on 3.0 is optimization, optimization, optimization. You know, we have bigger maps - gigantic maps - a lot more players and unfortunately a lot of our older systems ran these updates all of the time. Which means if we throw in 60 ships, that’s 60 big, gigantic IFCS updates which are heavy, physical beasts that eat up all of that yummy yummy CPU power. Obviously, if you have a ship way down there, you don’t care about him. You only care about the ships that are nearby that you’re combating or with your buddies, so we only want to update just those things and I put in something, it was actually during bugsmashers, to only show those ships that were nearby. Unfortunately, that had a knock-on side effect where on the server if you had an AI guy over here, he wouldn’t update and he couldn’t handle the mission. So, if you go fly over he would eventually tick on and then teleport away and it would cause havoc. So, I had to build upon that optimization so that it wasn’t just in-range but if you detected him on the sensor or on the dedicated server if it was a mission-critical thing he should do an update. So, we only update things we should update unless the server needs to do those special little things and WOOO performance and we get our fun CPU cycles.
One of the other things we were doing on optimization is we’re trying to get rid of a lot of our old systems. There was this thing called net serialize, this is basically the old way of the CryEngine to serialize a bunch of data. You basically had this function, it had a block of information, and you filled it all out but it doesn’t work well with our new system called serialized variables. The new way is multi-threaded, working in a batch, all of the great new stuff of the day. Net serialize - main thread, horrible flow setup. So, we’re trying to rip out that last thing. We only had three things left that used it, we ripped out two and we actually have a third one getting worked on right now so hopefully we can just toss away that old system and use that awesome new serialized variable flow and I think Clive will be very very happy once we finish that off.
Wayne Owen (WO): So, performance has gotten much better as of late. We’ve managed to increase the average frame rate in many places, even really busy places like Grim Hex and Levski. We’ve managed to get more people in - more clients in at the same time - we went to evocati with over 40 people not long ago and we’ve even run local tests using what we call headless clients, which is kind of what it sounds - it’s basically just an automated way of connecting clients - and we actually had I think ten or twelve PCs, we had each one running about seven to ten headless clients and we managed to connect over a hundred players - not sure of the exact number but managed to get over a hundred people in at the same time with stability. There’s an actual stable server with that many people in so that was really exciting for us to hit that.
General performance is better, even localized as a I say. We’ve gone through periods of time with a frame rate that’s been sub-30 which is when QA tends to get worried - that’s when we will start throwing bugs in if we’re seeing an average kind of FPS below 30 in any area and yet almost all of those areas are pushing well over 30 into the mid 40s, high 50s. It’s much much better performance throughout.
Steven Brennon (SB): Our priorities at the moment are to get a build that is stable but also has good player time features so we’re focusing on flight and traversal, planetary stuff, landing, missions, shopping, just to make sure all of these are functioning to get out to a PTU push in the hopes that, you know, it’s a good experience when we do go to PTU.
EKD: As you may recall, last week our production schedule showed that we were at 179 must fix issues towards the release of PTU across all categories. I’m thrilled to say that the result of all of the hard work by the team is that we’ve burned down enough issues to push 3.0 to PTU to start a larger round of testing. Now, from here we’ll be releasing updates to more and more PTU testers until we feel thorough improvements have been completed, then 3.0 will be released live. Now, don’t forget to check out our production schedule on Friday to see the progress we’re continuing to make and see you next week here on burndown.
SG: While there is no feature today, we do have some very special video content coming your way. Our anniversary special starts tomorrow and to celebrate the sale, we’ve recorded 8 episodes of Around the ‘Verse to profile all of the ship and vehicle manufacturers in-game and include a few surprises.
CR: Yes, there’ll be a few surprises so be sure to check out our website each and every day over the next week so you don’t miss an installment of the series or your chance to pick up a ship that is not normally available.
SG: Yes, that is right and thanks to all of our Subscribers and backers for your support. We couldn’t do any of this without you.
CR: Yes, we definitely couldn’t, so thank you very much. And for those of you in the United States, Happy Thanksgiving for those celebrating. And thanks to our dedicated testers out there playing the latest build and getting it closer to everyone’s hands in 3.0 Live.
SG: Yes, thank you - and until tomorrow we’ll see you…
Both: Around the ‘Verse!