[0:51] What are some of the frustrating sides of things for you as players?
Vincent is not fond of the holotable and has a team just for testing it out. Randy does not like the party system and hopes for the improvements to be implemented soon™
[1:46] What techniques do you recommend for intrepid bug hunters? How can we most efficiently collect the information you need?
Be clear and concise.Provide steps running through what you started doing. Include both what you were doing when the issue arose and what you were doing before it as well.
[3:17] Do you use Issue Council to understand problems better?
Yes, especially in LA and UK offices.
[4:29] Does causing chaos and havoc on the PTU help test? Are the QA targets shifting constantly with the flux of new patches to the PTU?
Even just logging in for a short time helps. However if you are planning something that will involve other people then make sure you have their permission. QA targets are in an unstable environment to keep up with the new mechanics and content but provide a focused intense service.
[8:17] How is your time spent when doing QA work?
40% regression, 40% targetted, 30% ad-hoc testing. 110% cause QA are awesome.
[10:24] When testing, how often are past bugs retested?
Only when areas covering the past bugs are redone. If landing gear gets a code change, then we look quickly at the bugs for landing gear. Otherwise, we just look back at previous fixes if a new bug is found.
[11:58] How is this feedback communicated to and used by the rest of the QA and the designers? What can we do to make the feedback more useful?
The community gave us so much feedback it was almost information overload so we’re going to slow down the feedback so designers have more time to parse it. And we’re going to try to make it easier to digest when it does come in. You should keep things clear, concise and within scope but don’t be afraid to give examples.
[15:53] What plans do you have for yourself in the PU?
Randy wants to do mercenary/pirate or a miner. Vincent initial mission is to hunt Randy Vasquez down and kill him. Then he either wants to be a CIG bounty hunter or information runner, sell location information on other CIG people so others can hunt them down.
[18:14] What are the priorities of development among CIG? e.g. Ships, Characters, System, etc.
CIG is big and so everything is split up into hierarchies that have their own setup if you will. They have their tasks that they do and for the most part remain that way. Priorities change almost daily and sometimes hourly so we don’t have a given priority always. We don’t however pull Artists to help with an Engineer job, vice versa, only the appropriate people assist where they’re needed in order to maintain efficiency. There isn’t a given task that’s above one another because of priorities constantly shifting.
[21:09] What are Randy Vasquez’ goals for the year? What new features may we see soon?
To survive as a producer but he’s enjoying the change so far. Shields were just given an audit. They had to look at how lasers affect them, kinetics etc. As well as take into account their functional role. These changes are also being on for armour. So we may see the new damage system soon.
Randy Vasquez (RV): Hey guys, my names Randy Vasquez, Associate Producer for Cloud Imperium Games. We’re here for 10 for the Developers and for the first time we have…
Vincent Sinatra (VS): Vincent Sinatra, QA Lead.
RV: Nice. So we have questions, but we also want to thank subscribers for making this possible. It’s really awesome that we get to kind of directly interact with you guys as well as take questions and then kind of give you our opinions or suggestions or kind of our ideas and thoughts on what you guys are asking so, ready to get started?
VS: Oh yeah, let’s do it.
Solitude asks: For both – What aspects of the gamer process (by which I mean setting resolution, visual quality, audio levels, control schema, etc) do you find frustrating // would you change given scope and resource to do so?
VS: For me that answer’s easy it’s the holo table RV: What? Everybody loves the holotable VS: Really? Who are these people? VS: I don’t know VS: I personally don’t really like the way it’s functioning currently. I don’t like to look at it, don’t like to test it. In fact I got some new testers now we’re training up and i’m making them test it because they’re the nuggets and that’s their job. RV: I guess personally the party system right now. The party system bothers me but at the same time is I know that these are being worked on and a lot of the feedback from the fans. We’ve taken tonnes of feedback on things and, so we’re definitely working on it. So we’re pushing that soon. I cannot say when so do not ask when but if you’re even thinking about asking don’t even do it! VS: Yeah pretty much!
Amontillado asks: What sort of techniques or procedures do you recommend for those of us that want to hunt and report bugs to CIG? How can we most efficiently go about collecting the information that you need?
RV: So this is kind of a question for you … VS: Yeah, I believe this is probably a QA question and Issue Council type question. So basically what we primarily want is people to be concise, clear, clear steps running through things you started doing but also include … sometimes you’re only reporting the thing that you saw and what you were doing when you saw it, but sometimes what you were doing before that impact things. Like we had certain situations where people were getting disconnected while joining Crusader they would just tell us “hey, I got disconnected when joining Crusader” but, you know what, it was because they were joining Arena Commander before they transitioned to Crusader and their tokens weren’t getting cleared and that was the issue. So as much information as possible they can provide is usually the best for us because then we can pick through all that information and get really good steps for the developer so they can actually get a fix in. RV: So I know with like release notes and stuff we have … do you guys set the release notes? VS: Yeah. RV: So maybe in there we can be like “hey guys, can you focus on this?” Do we do that right now? VS: We don’t do that right now. That’s not actually a bad idea. You know if somebody was a Producer … RV: What? VS: … they might be able to push something like that forward! That might be a good idea going forward. RV: Man, see this is what happens when I open my mouth! VS: Yeah. Exactly!
Neuroquila asks: Since, thanks to the Issue council, players are somewhat involved in the QA process, do you search the council frequently in search of hints to better explain what went wrong for a particular problem or do you just wait for issues to receive the appropriate number of votes and then react?
VS: Well, me personally I’m in LA and we’ve got a really small team, mostly we’re embedded testers, we do a lot helping the developers and stuff like that, I know our Austin and UK we’ve got great QA teams out there who are always in the Issue Council making sure they’re checking issues. Now a lot of those issues are already written up in our database cause we ran into them during our normal sanity checks and smoke tests and sweeps and stuff like that, but the ones that do get rise to the top, those are the ones we investigate, we make sure we get in the database, we make sure we get fixes out there because we know the backers want that stuff fixed as soon as possible. RV: So some people might not even know what the Issue Council is. What, like… VS: Issue Council is where the backers and all our Subscribers should be going to report any sort of issues, and then if you run into it you vote it up and the higher it goes in the chain the sooner we take a look at it and get it taken care of. RV: Nice. That’s kind of cool. I’ve never used it since I just kind of go over to your desk and I’m like… VS: Yeah, you say is this happening and then you hit me and yeah, that’s exactly what happens pretty much for everybody RV: Basically. Poor guy has to put up with this.
Drugh asks: How can we support testing better. Should we just try more crazy stuff, actually intentionally cause havoc to break things? Does the wealth of PTU data provide any new insight into the process? How does one prepare to QA the sheer number of future features while simultaneously changing the engine, adding new part, increasing the complexity. I feel this must be a moving target for QA in an unstable environment?
RV: Yes. VS: Yes. RV: Yes. RV: Yes. It is definitely an unstable environment. In fact more days than not it is quite unstable. There’s going to be certain areas in the code you can’t even look at. There’s new iterations coming down. As far as causing havoc, well when you’re out there in the PTU I think we want more controlled havoc… RV: Actually yeah. We would say controlled havoc or kind of like planned chaos VS: Planned chaos RV: That was the one we were kind of like.. VS: Planned chaos that’s what we want. Because doing strange and weird things is always great. Sometimes you find the best bugs by doing something that the average user would never even conceive of. Like what happens if I land this ship of this ship on top of another ship and you end up with some sort of shipception or something like that. Sometimes you can do that sort of wacky stuff and you could find some really good issues. But you don’t want to do that to people who don’t know what you’re up to. Then it’s just griefing. RV: Basically the point; don’t be a griefer don’t be dicks. Just be nice to people and if you want to test something crazy seriously just… We were talking about this, just be like… VS: Just ask people RV: Be like “Hey guys I want to do something crazy. Is anyone down for crazy?”. VS: Yeah! RV: Usually people will be like “Oh crazy? We’re in, we’re doing this!” VS: Yeah. We got a pretty good crazy base going on! People are just like to say “Hey I want to try something wacky today does anybody else want to come here and help me out with this?” And usually we get one or two people to jump in. I know i’ve done it a live server a couple of times. So that’s one of those things that you can do as far as havoc. As far as the PTU and how it helps us with the process. It really helps when it comes to performance, stability. We get those metrics from hundreds, thousands of users and obviously that’s something we can’t do internally because our Test Teams are only so big RV: And also on that. People need to just log in and just be there. Cause even just logging in for a little bit here and there so that we can put the servers through their paces, the Engineers are getting really good information The Designers are getting good information because the Designer can design something and then all of a sudden it works when you have one or two people there and you see this all the time probably. Five people there, but as soon as you hit the ninth or the tenth person everything just breaks down like crazy VS: Whereever you can get those numbers it really helps. Because again the QA Teams are kept small and focused so we can you know ‘laser fine accuracy’ on what we need to be checking but you know… RV: SO point take away is: If you want to do some crazy stuff seriously just put it in chat and be like “Hey you guys i’m trying this out does anyone want to join me?” VS: Ask permission before you start using the Glaive to lift up other ships and stuff like that RV: Just don’t kill someone cause we’ve seen things on Youtube where people are just going crazy and just ramming another ship and the person’s not even taken off. And i’m like “Come on people. You know that’s griefing”. VS: Because you don’t know how long that person has to play. You may be in there for a four house session trying to hit up as much stuff as you possibly can. They’re only there for fifteen minutes. They got kids, they’ve got to change diapers, they got to get the food ready, they got a bunch of other stuff. They just want to jump in and see what we have on the next PTU patch and they don’t necessarily want to help you do crazy stuff. So just clear it ahead of time if you’re going to involve other people in your wacky havoc causing tests. RV: The other thing is. We have such a strong community we don’t need stuff like that it would just bring it down. Just help each other out. Controlled and planned chaos is where it’s at.
Logical Chimp asks: When looking at the QA team work load, roughly what is the split in testing between: Running through pre-defined scripts/regression testing. Adhoc/experimental testing. Targeted/specific feature testing (I.E gathering stats on ship performance). As a follow-on, what determines/changes this allocation (I.E what determines if you need to increase time on regression or spend more time on ad-hoc testing, etc)?
VS: Well, as we’ve mentioned, it’s largely a moving target. You know, usually we have our set predetermined tests that we’re going to do on every single build be it a smoke or a sandy check or something like that, and whenever a new feature is developed of course we’re going to go through a gambit of tests that are designed around that feature, but we’re also going to ad-hoc areas around that feature, you know as it comes out. I would say it’s probably about 40% regression, 40% targetted, 30% ad-hoc, because you know QA, we give 110% every time. RV: 110% VS: Yes, that’s right. RV: So actually we were talking about this earlier, cause like whenever bugs come in and, so we have a little thread in production that’s like hey these are being flared out, and it’s mostly just production and QA that are on these threads, and it’s like hey, this is the latest blocker, this is the latest thing, whether it’s found through the community or whether it’s found just through us internally testing and all that stuff, these are getting major priority, so as soon as these are sent out the developers get it, and once the developers fix it, a producer’s basically hitting up one of the leads like you saying hey, we just fixed this, here’s the change list… VS: Lets get some eyes on it, let’s get some bodies on it… RV: Exactly, and then people will just change focus like that and just be like alright, we need this, we need this now, we need this now, so yeah, it’s like building in quicksand… VS: Pretty much, the situation is always fluid, from one day to the next, from one hour to the next, cause all of a sudden someone will come along with something new, and that all of a sudden takes priority and everything else that you had gets shifted down a rung on the ladder and you have to get to it later. RV: Yeah, it’s always a lot of fun. VS: Fun is not quite the… no actually it is fun. RV: It’s a lot of fun. VS: It’s a lot of fun. RV: Alright, so next question is from Steve Hunter…
Steve Hunter asks: When doing QA testing, past resolved bugs are often retested to make sure that new fixes don’t cause them to reoccur. As this would make the test list grow uncontrollably large quickly, how do you decide when to no longer retest an old bug? And thanks for all the good work.
RV: Thanks Steve! VS: This is actually a really good question. So when it comes to that, usually we focus on areas that have been affected by recent code changes, like recently Mark McCall came to me said hey, we’ve done a bunch of changes to landing gear, so that immediately tasks us out to start focusing, checking all those old landing gear issues to make sure that those don’t recur, because you know of the changes that he just implemented, to make sure that previous fixes didn’t get stomped on by new changelists and so on. So i mean, that’s pretty much, we try to stay focused on the things that’re in motion and are still changing, and then if something hasn’t changed build after build after build, we’re not necessarily going to be looking at all those old issues again. If they do return we’ll reopen them, but we don’t necessarily focus on them. RV: Yeah and that’s the thing, it’s like I know, once the bugs have been through their paces, they’ve been resolved and fixed, once it hits a new area if we see it again we’ll check the database and be like, okay, this is back, and in all honesty there’ve been some bugs, especially with landing gear in particular that the landing gear comes back so many times VS: That’s why I brought it up, cause poor Mark has been banging his head against the wall, I swear to god some of these bugs spend time in the Lazarus pit and rise up stronger than they were before, it’s just unbelievable with some of them. RV: It is pretty crazy.
Far-Seeker asks: I know you have posted some about this in your feedback threads already, but could you give us an overview of this activity, especially anything that submitters can do to make the feedback more useful, as well as more details on how this feedback will be communicated to and used by the rest of the QA and the ship designers?
VS: Yeah, so basically what I’m trying to do is, because it seems like we are getting so much feedback, I mean I had pages and pages and pages of feedback … RV: Which sounds pretty awesome by the way. VS: It was very awesome. I mean the community is very passionate and they gave us just tonnes of information but some of it was information overload for developers and designers who are already working hard on new things. And they’re like “When am I going to get time to parse through all this data?” So moving forward we’re going to try to slow down the rate of those so that we give them more time with each ship for the most part. RV: It’s still in the works like when we’re actually going to be doing these I think, right? We still need numbers … VS: Yeah, we’re still breaking it down. Like I said, I was rapid fire, machinegun style getting that stuff out. And now obviously that was a little too much, so we’re going to try and slow it down a little bit. RV: Ambitions, rambunctious I might add. VS: Yes, yes. A little bit. A little bit overeager. I’m kind of new here, trying to make my mark and I’ve got to impress guys like Randy every other day. RV: It’s true. It’s true. VS: I was trying to work on that. But as far as what details, just keep things concise; keep things clear. Try to keep things within scope: we’re not always looking for “Hey, please buff my favourite ship” type of stuff. You want to make sure you know exactly the type of thing we’re looking for. Perhaps on my part I need to be a little more clear as to what we’re asking for. Now we’ve even had some growing pains through some of the threads wit that … RV: Yeah. That’s the thing with some of the other stuff we were talking about earlier is I would love for people to break down the feedback from different sections. VS: Definitely. RV: So even multiple post so that way like “Hey, this is the arts posts on this”. So that way you give everything for art, break it down to design, mechanics … VS: Exactly. Mechanics, balance … RV: … to break it down as much as possible. Yeah, balance-wise like “Alright so I’m flying a 300, I’m flying a 315, they’re almost the same ship but not really”. But then just breaking it down that thing like as much as … don’t be as verbose. Clear, concise: definitely. But don’t be afraid to give more examples or give, at least … because the examples you guys give, as well as just the detail you guys can give us, anything is useful. VS: Yeah and I was pretty much amazed at the detail because we had graphics … RV: Yeah! VS: Particularly for the 300, people were like “Hey this is how the redesign could totally work” and I’m like “Is this a resume? Is this a job application? Or is this just normal feedback?”. So the detail was crazy good in some regards and then in others people are all like “Hey, give me a buff”. For the most part. I want, I want … RV: Make this ship the most powerful. VS: … I want five gimbals on my ship! Or something like that. Those we don’t … RV: I love those gimbals. VS: Hey who doesn’t? But for the most part as long as it is clear, as long as it is concise. Keep things polite in the threads. If somebody says something you don’t agree with, hey we’re all trying to make the best game together. That’s the great thing about our community is we’re working together. RV: And honestly, the pointing out that thing, if you guys have things different, varying opinions that is perfectly fine. We actually welcome that because then we see different perspectives on how people are looking at the designs, how people are looking at the ships, the role of the ship, everything. Any feedback is good feedback. VS: Yeah. RV: We can’t always act on everything and we’re not going to be acting on it instantly so give us time because we’re still working on things. We only have so many people to actually do everything. VS: Yeah. And I’m going to try and make it easier for the devs and designers to digest when it does come in. Try to get a top 5 bullet points and “Here’s the rest of the report when you guy get around to it but these are major things that are either really bothering the community or they’re really excited about getting implemented for their particular favourite ships”. RV: It would be super useful. VS: Yeah. Very useful. All of it.
AragornBH asks: Since you have played many hours of Star Citizen and know the games mechanics so well, what plans do you have for yourself in the PU as a gamer?
RV: So this is kind of a loaded question and the reason why it’s loaded is because I really want to have kind of two personas when I go out there. I want to do one where I’m a mercenary/pirate and stuff like that and seriously just get in that whole clandestine thing and like be on that black market. I think that would be hilariously fun cause I never play that kind of character. Usually when I play an online game I’m either support or a healer. Of course the other one I would play would be a miner. Like when I played Eve, I was all about being a miner and a trader. So I think that would be really cool. Just doing some mining, doing some cargo stuff, that’s why I like when I was given the ship the Caterpillar… I was like all about it cause it had that different mix of things that I was like, ‘I can do this’… VS: There you go. RV: I can do this. I’m actually looking really forward to just.. VS: That sounds pretty good. RV: It’s going to change. VS: Like you said everything is like a moving target but yeah that’s pretty good. I know once the game is out and I have one mission in mind and that is to find Randy Vasquez and kill him. Pretty much. I mean, I like cutting you guys down, that’s the most fun of the day… RV: I hate you in play test. VS: Completely destroying you guys in play test is so much fun. RV: I can’t stand you so much in play test. VS: So once we’re actually out there I figure I could be some sort of CIG bounty hunter and I’m just going around letting other people know where CIG people are hanging out… RV: Hunting CIG people. VS: Maybe I can do some screen hacking. Then you know, “hey guys, they’re over here” and we swoop in and destroy you all. RV: I think we did a play test what is it, a month or two ago, and basically I think it was McCall or might have been you I think was like, ‘by the way guys, Vasquez is at Port Kariya’. VS: Oh yeah. RV: Next thing I know I had like four or five ships coming in and I was like, ‘oh god’ and then I’m trying to hide from everybody. I shot at a few people but I think I shot Jeremiah. VS: That’s what I’ll be, I’ll be an information broker who then leaks out where CIG’s people are and then people can go hunt them and have fun doing that. RV: Nice, nice. VS: Make a little you know… RV/VS: Money on the side. VS: A little bit. A QA salary isn’t exactly that great.
Space Is Nice asks: What is the priority of developing things? Is it ships -> places -> economy. What takes priority over other things, like shipping over ship buying or more to do in one place over more places to go? Thanks!
RV: Alright so. As we’ve stated a few times, development is always a moving target and priorities shift daily, if not sometimes hourly based on what’s broken or what the fire is and stuff because we want to give you guys a stable game VS: Yeah RV: We need to give you guys, also QA a stable game because… VS: Stable builds. RV: You guys can’t test if it’s not stable VS: Exactly RV: You guys testing in unstable environments doesn’t help us because we don’t know if the bugs you’re giving us are actually there. VS: And we can’t help you because someone will be like “hey will you check this,” and it’s like, “Well we can’t check it because…” RV: So kind of to go off the questions like. Versus ships, versus levels, versus economy, PU, mechanics, systems; we have so many people around the globe working on this game. Everyone has their own focus, everyone has their own priorities. So we have an overall global organization and then we have within that organization we have the different studios and then within that we have the each individual teams were on something where they’re Designers, they’re Artists, QA, Production, Engineers, Audio, Narrative. We have so many different aspects of this that everyone is just all over the place all the time. There’s no ever a priority one thing over another. Like for instance if Crusader is broken, then that’s kind of a level design but also design, but it’s engineering, not so much Art so that then becomes a focus for the select people. VS: Where as the Artists will still be working on the stuff that they’re already working on. RV: Yeah, yup so they don’t change focus. Now if a ship is completely broken and a ship needs to be redone due to this area, whether they need to be redone for LOD’s ship setup issues. VS: Or a ship setup issue, LOD’s. RV: Then it’s like the tech designers or artist, they’re shifting over to help with this and sometimes they need engineer support. It is a case by case basis on really what the priority is and that is actually coming down from the community because every time the community, like they’re yelling for something, they need something, they want something, we’re taking that into account. Also the leadership team of CIG is also like “Hey, so.” They’re kind of always playing the balance game of, “Alright this is what the community wants and this is what they need, this is the leadership team wants and needs,” and kind of just doing that balance game of, “This is our priority for right now,” and it’s our priority until something else changes and we still have like schedules that we’re working on, but then VS: We still have planned feature rollouts and stuff like that as well RV: Yeah basically. VS: It’s just like you said things kind of change up midstream every now and then. RV: Yup, it is what it is.
Daz asks: Hi Randy, can you share some of both your own goals and the company goals in production this year. Is there a particular area you love working on and are keen on improving? And what games features are we likely to see in the short, medium term this year?
RV: So this one for me myself I just want to survive as a Producer. It is so different from a Designer. I’m enjoying it, it’s completely different. I’m really loving, I like the communication aspect of it. VS: Yeah. Yeah you’re really good at communicating. Really good at communicating. I got an inbox full of your communication. RV: So I love that and I love facilitation. So basically making sure the team is taken care of. going to everybody making sure they got everything they need “Hey what’s up what do you guys need?” VS: Backrubs. Regular backrubs, come to Randy RV: Nice to know my numbers you may have saw on the bathroom wall thanks VS: That could be a good and a bad thing RV: So i’m really enjoying that but what I want to improve on though. I really want to get the ships pushed out. In a better timetable, I know we’re, we push ships out and we’re letting you guys know when ships are hitting. And each ship is its own little special snowflake. So it’s been hard to nail that down because each ship has its own mechanics, each ship has its own features VS: Then there’s those features that’s tied to each of the ships. That need to be pushed out stuff like that. Like you said the shields and armour balance that Callix and everybody has been working really hard on RV: Callix just finished that audit for the shields VS: Just finished that whole audit for the shields RV: And you had to do all the testing too VS: Yea I had to all the tests. Trying to get all those out would be really good and help give us more a final picture of what the game’s going to be like, eventually. With each ship that comes down it’s alright but that’s just a small piece of the greater puzzle RV: And that’s the thing. The audit you just worked on is part of something bigger too because we’re doing, we’re redoing doing some of the shields and how they work. Just cause it’s going to help optimise the game overall. VS: And how lasers affect the shields, how ballistics affect the shields RV: All that stuff is just being taken into consideration. So the design’s changing a little bit. And it’s changing for the better and it’s changing to make sure that players are given the proper feedback they need. As well as it has all the functionality necessary things it has to have. Shields are just crazy integral to the whole thing I think as a game. Especially for dog fighting. VS: Especially for dog fighting. Well I mean it’s hard to balance the ships against each other when that system isn’t fully fleshed out just yet. RV: Where you around still when they had the Constellation and the Hornet bug? VS: Yeah, yeah, yeah. I was one of the ones, the Hornet and the Constellation were using the exact same shields. RV: That was crazy VS: That was a fun one RV: There’s so many things with shields. So yeah that’s a good example of something we’re working on that’s going to be coming soon VS: Yep.
VS: So that’s 10 for the Developers for us today. Once again want to thank all the subscribers and all the backers, everybody who makes this possible for us to do this.
RV: Thank you so much.
VS: I know maybe we weren’t quite as entertaining as Tracy and Bender.
RV: We cannot follow…
VS: We were trying and would like to come back and if you guys are into it.
RV: If they’re into it.
VS: If they’re into it, of course, everything has to be consensual but you know… if we could come back it might be nice.
RV: Yeah, it would be awesome. I always love talking to the community.
VS: I always like this opportunity, this is definitely something new for me and I’m enjoying it and you know, hitting the ground running and hoping.
RV: We’ll communicate later.
VS: We’ll communicate later, yeah great. Thank you, more communication.