DU015 – Guerrilla User Testing

Transcript

Episode – DU-015 Guerrilla User Testing Hosts: Chris Meirs and Carla Lindarte

00:17 Chris: Hello and welcome to Design Untangle with me Chris Mears and sitting over there is Carla Lindarte. How’s it going?

00:25 Carla: Hello Chris. How you doing?

00:27 Chris: Yeah, very good too. We using our new podcast recording tool today, so hopefully it doesn’t screw up.

00:34 Carla: Yeah, hopefully it works. How are you doing?

00:40 Chris: Yeah, I’m good. I had a massage a few hours ago, so I’m very oily at the moment.

00:46 Carla: I couldn’t believe, I also had a massage at work today.

00:49 Chris: Oh, JJ. Maybe it was the same person. We don’t have the message together just in case anyone was wondering.

00:56 Carla: No, no, no. I had it at work. It was really, really, really good. So I’m really relaxed. Ready for the weekend with these horrible weather.

01:05 Chris: Yes. Spring has finished already. We’re back into winter, there you go. That’s the joys of living in the U.K.

01:13 Carla: So today we are going to talk about guerrilla testing, right?

01:18 Chris: Yeah, that was what I thought we were talking about anyway. Okay. What the hell is guerrilla testing Carla?

01:25 Carla: What is it? It’s just like random testing. No, it is not. Guerrilla testing is basically, it is more related to the way you recruit participants rather than lack of

1

planning or random research, if that makes sense. So basically means that you go to a place where you think you’d be more likely to find the users you’re looking for. And randomly select people you know, either is it on the street or even in a party or in a bar or a restaurant, wherever you think your target audience is and try to talk to them, without previously, setting up the whole research. So that is in my opinion, what a guerrilla testing is. But what do you think it is, Chris?

02:17 Chris: Yeah, that sounds about right. I’d say it is one of the scarier things you can do as a UX designer or researcher because you’re basically going up and talking to complete strangers. Generally speaking, it can be billed as sort of like you say, a a half- assed version of research when you haven’t got time to do it properly. But that’s not really the case. It’s just a technique that you can have in your arsenal to gather data from users when, you can’t set up a proper lab session or observed them in the context of their jobs or whatever it is you’re looking at. So it’s don’t think of it as third-class methodology. It’s just a different way of gathering insight.

03:05 Carla: Exactly. So it’s also called spot testing I think popup testing. I’ve met many teams call it a different thing, but it also, I think we have to highlight the fact that it is important. Guerrilla testing is actually useful because you can see the user in the context of where they actually going to start interacting with the product or that you are actually designing. So that is a good thing as sometimes when you put people in a lab environment is not really a real scenario. Whereas it really has the advantage of being, at least closer to the real experience that the user’s going to have.

03:48 Chris: Well in some contexts anyway, if you’re designing an app, but then certainly it’s probably more likely they’re going to be doing that kind of out and about. It can obviously be different if you’re designing something that they would do at their, desk basically in their day job. And so it depends on the project how valid the context is I think. But in any case, you can usually get some sort of useful feedback from people just from a usability point of view if nothing else.

04:20 Carla: I mean, it’s also very like low cost, way of getting user of feedback. I mean, if you’re in a team where they’re saying, no, we’re not going to do any research. We don’t have money to do any testing at the moment. And you can always suggest, doing some guerrilla testing cause some research is better than none. However, it doesn’t mean you have to have less rigor in the, in the research. And what I mean with that is that you still need to have a clear view of what are the things that you want to learn, what are the hypotheses that you testing and, a clear view of what kind of questions you’re going to be asking, et cetera. So you still need to plan for it. And you need to have very clear in mind, what is this thing that you want to learn from that particular research activity? Because I’ve seen, teams in the past, that they just go off and then they start stuffing people and they just started just asking random questions without a particular structure. They don’t really know what they’re trying to do. Sometimes they ask oh, do you like this? And that’s obviously a very bad question to ask. So you still need to have a more clear view of what you want to achieve and have prepared discussion guide and all that sort of stuff.

05:44 Chris: I’ve kind of seen it. The client will be just like, oh, just go and gorilla test it. That you have to kind of push back on that a little bit and say, okay fine, we can do that, but what are we trying to achieve? If what we are trying to learn? Often it’s seen as just a way of ticking the user research box without spending any money. But I think it’s your job as UX designers to make them aware that, as Carla says, there’s still planning involved as still kind of understanding what the value of doing that gorilla research, is. It going to answer any questions? It should never just be okay. We took it out and showed it. Some users you read showed it six users, happy days, let’s go to the pub. There should actually be a point to it. You should get or at least be aiming to get some value out of it. Don’t just do it for the sake of it essentially.

06:37 Carla: Yeah, exactly right. So not just do it for the sake of it and just do it properly. And so you still need to think about, the location where you’re going to go. You have to put together a plan, you have to have in mind as well. How many people are you targeting? So how many people you think is kind of a success, if I go, for half a day or for a few hours to this particular branch, or store, or whatever, define users. You also have something that I’ve learned as well, throughout like doing guerrilla research in the past is that if you go with too many questions in mind, I don’t think it’s going to work. And what I mean with that is that you call it, just go on and try to get, a lot of time with someone with one person, and just, spend hours with them, like trying to understand so many different things. You have to make it quick and you have to make it simple. It is better to run late to do various, testing sessions rather than trying to cover way too much with one user.

07:51 Chris: Yeah. You’ve got to remember what you’re doing here, right? You’re approaching a stranger asking for their time. Very often. They’re not particularly going to be keen on that, because I think you’re trying to sell them something or whatever. Say it’s unrealistic to have a hour long lab stow discussion guys and expect to get through all of that. So you have to be really targeted in what you’re trying to find out before they, are she on your way or kick you up the ass or punchy.

08:23 Carla: And also I’ve also seen people using guerrilla testing as a way of proving something. And what I mean with that is that they might have. Let’s just, interview, you know, let’s go do some Gary Lettuce thing. Let’s find 10 people and see if they like this button. Right? And then this out. Yeah. Eight out of 10, like the button. So we going ahead with the button. That is not the way or how or why you do gorilla testing. So you, if you have like a quantitative needs, I would suggest different method. If you are to add testing accessibility or actual usability, I don’t think is the right method either. I think it’s also when you try and to test whether or not, you are proving a proposition like, is this a good product, is this something that we, our company should invest.

09:18 Chris: I don’t think guerrilla testing is the right choice either. So you just need to be aware of the benefits of guerrilla testing but also why you shouldn’t be, using that method. I’ve also seen teams doing like what they call AB testing and I actually would like to get your opinion in that, Chris. Cause I seen people, with for example, as some mobile screens and they have in one screen this, a different design than the other and they asked people which one do you like best? And that’s what they call AB testing for me. That’s not AB testing for me. AB testing is more like a quantitative, mostly variants test. Kind of like, method. But, they use that to say, do you prefer this or that? What, what’s your opinion on that case?

10:06 Chris: People were pretty bad at predicting future behaviour and the only reason really you want to compare two designs is to see if one drives a particular behaviour better or more effective than another one. And you’re not going to be able to get that information by saying, oh, which ones you like better? This one or this one that’s not going to prove if anyone will actually con, their behaviour will follow what you want it to one way or the other. You’re only going to be able to do that on an AB test with kind of large volume and get statistical significance on that. You might be able to spot some usability issues with a certain approach, which is fine. But if you’re ever asking someone, do you like this one or this one in any sort of test, whether it’s guerilla or otherwise, then you need to revisit your discussion guide.

11:00 Carla: Yeah, exactly. I have to say my opinion, but I just wanted to make sure I wasn’t completely crazy. So what is a discussion guide then?

11:11 Chris: well that’s a bigger question than just gorilla testing. That’s why research thing. But essentially it is a framework of the conversation is basically a prompter for whoever is running the research to remember what they will need to find out and generally how the conversation will flow. I use Simonia at a very high level when I’m doing testing because I like to let the conversation flow a little bit is going to be a bit lame if you’re literally reading Oh, say when do you use this? How many hours a day? What’s your favourite color? Especially in a guerilla setting where it’s very informal because you’re approaching complete strangers. They’re not expecting to be part of your user research. It needs to just act as a kind of reminder of a prompter. And to be honest, I find that very often I don’t even look at it. I’m familiar in my mind just from writing it beforehand.

12:14 Carla: Right. That is, a very, very good point. I also think that the discussion guys are important. If you have a team working with you as well. So I’ve been in situations where we say, okay, let’s divide and conquer. You take that side of the shop. I take this out at the shop and then we go and normally in pairs, because you will be good to always have someone observing and taking notes whilst you conduct the session if possible. And it also helps when you know for note-taking on the structure to have some kind of common themes. And as you said, it doesn’t have to be prescriptive but it gives you some structure for, you know, what is it that you’d want to achieve and then it’s easier to go back and kind of get the learnings consistently across all the sessions

13:03 Carla: So what else do we need? Do people need before the session? So getting at discussion guide, also a consent form. I particularly like getting consent forms. Some people don’t do it, but I think it is important for people to understand that, they’re being part of this research fees and they signing for you to be able to use, the material in any, particular way. So there’s lots of templates, online that you can use and tailored to your company or your project.

13:37 Chris: So consent is, an interesting one and something that’s going to become much more important in the coming month when a piece of legislation called GDPR comes in that is all around privacy and data and how your data is used. So you need to make it very clear why you’re speaking to this person. If recording it was going to happen. So that recording, what can they do if they want to have that recording removed at a later date or deleted? I would recommend not keeping any personal data or archives of research you’ve done any longer, then you absolutely have to, you properly. If you have one at your company legal department and they want to just check over that consent form, that’s kind of obviously with all the Facebook stuff that’s been happening and these new laws coming in, you really need to make sure that people are actually agreeing to give you information and understand what they information is being given for. Even if it is just a guerrilla style usability test.

14:42 Carla: Definitely that. And it couldn’t be, it can’t be, this may say these because of GDPR and, and it’s also like ethically is a good thing too, as an interviewer to be able to tell people, this is how we are going to use your data. And then please sign here and give us consent in the way we’re going to use it.

15:05 Chris: Yes, because the example I’ve seen as you might think, oh who cares is just a chat about some screens or whatever. But you know, if you’ve got a recording of someone taking the piss out of a certain design or laughing at it, or whether a new show that a big company meeting, you’ve then put this person in kind of a particular position where they’re being shown to be like mocking or an idiot or whatever. And you just have to think about the implications of how that recording can be used.

15:38 Carla: Yeah, exactly. So that consent form is really, really important. You can have it printed or in the past I’ve also done digital ones like, Survey Monkey stuff that records all of the other in mind that even though you’re going to be like on the go and in a public space, Norman e you still need to kind of find a space where you could have more quiet conversations with people. I mean we used to, when I used to do research in stores, it’s like finding that space where you can ask people to kind of go aside and have conversations. So before you start any, research plan, do you need to think about the location you go in, whether or not you have permissions to do it, to be there first of all and to be able to record or take pictures of whatever you want to do in that particular location. And also having a like a space or a station, where you could actually take the person, take them through any prototypes or any visual [inaudible 00:16:51].

16:51 Carla: You don’t like that [inaudible 00:16:56].
16:56 Chris: That’s really good. I can do that now just whenever. I don’t know what you’re saying. I’ll just go…

17:08 Carla: So the location is really important and I’m having permissions to be there and also finding a space and you can get your users. Then maybe if you planning to do it for more than 20 minutes, at least you can sit down or leave if they carrying bags or whatever. They have a space to do that. I know it’s silly, but from experience, I can tell you that, it could get really complicated. People might be able to help you out and want to help you out, but if you don’t give them, enough space or time or whatever it might, they might say no. So you have to be ready for all of these things. Also, they are ready with charges. They can show you’re going to have WiFi, cause sometimes you don’t have access to WiFi and if you needed, you can carry and carry something with you and devise with WiFi or you need it. All these things like to just, create a check list of all the things that you need and be ready if it’s raining, if it’s going to rain, check the weather. All of that is really important cause, depending on that, you’re going to get more or less users.

18:13 Chris: And there’s certain places you kind of learn as you go along I think, but where people may be a bit more willing to talk to you. So if you approach someone and they’re on their own and whatever it is, a station or something, you’re generally whereas very good results even. Whereas if it’s kind of a lot of people around and they’re waiting for something like train time to kill, you might get a slightly better result. And so, so what about just having that they have awareness about the environment and people’s body language and if they’re sitting there reading a book or you know, or they’re don’t want to be disturbed, then just don’t disturb them. You’ll just pick it up. I think as you do more of this, it can be quite intimidating at first.

19:08 Chris: Expects a lot of nos or weird looks. You get over that pretty quickly. I think as you approach more people, I remember I had to talk to various people about buses in Glasgow awhile ago. Surprisingly, it wasn’t as bad as I thought it would be that there were definitely some very suspicious people. Why the hell were you asking me about buses? I questioned that myself for little bit. Reassess my life. Yeah, you will get rejected quite a lot but just keep at it, you’ll get over it pretty quickly and remember to just take breaks every few hours because it is quite emotionally draining as well as the other thing I found.

19:53 Carla: Yeah it is. It’s very emotionally drained but, I’d like incentives is actually a very good way to attract people. So if you were with a retail company and you go into the shops, tried to get some like vouchers, so 10% discount or, when you targeting very like niche markets like parents for example, you can get them tickets for them to go to their kids somewhere; or just try to, depending on the type of users and that’s what is really important. Going back to what is it that you’re trying to achieve with your guerrilla testing and what kind of people I need for these particular hypotheses. I want to prove you can tailor your incentives, and make them more attractive for them. I remember once I saw a group of people, I’m near, we work in a mall gait doing some research. I think it was a health app or something like that. So they had, they were looking for health-conscious people and they were giving away like smoothies and protein bars and things like that. So then you kind of try to tailor what you given them with the type of users that you’re looking for.

21:05 Chris: Yeah, absolutely. I remember when I was doing the bus when I just bribed people with new cars.

21:13 Carla: Oh that’s good. So wow. I wish it wasn’t that resarch.

21:17 Chris: Yeah, it had a big budget for that one

21:19 Carla: Yeah. Also that something that we forgot like having an id or showing that you actually are a serious person doing a piece of research. That’s a very good idea. Serious. Like sometimes I confuse you with people like asking for charity or something like that. Because you don’t know, like there’s a lot of people on the street dying to talk to you. Oh, I’m not going to talk to you. Especially if you’re in London. No one wants to talk to you. But if you want to, if you can identify yourself, say look at wherever this design agency or I do this, just want a destined af with you. So having an id or something that supports that stories are also very important.

22:01 Chris: Yeah. And that’s true. I did get asked, who do I work for or whatever. So having some proof that you do actually work for a company and not to some weirdo talking to people at bus stops, was almost the case there. That’s definitely a good thing.

22:17 Carla:: Yeah. Yeah. Anything else about gorilla testing

22:23 Chris: No, I don’t think so. we’ve got quite a good blog article on the UXReview.co.uk about gorilla testing and some tips about how to do that. Some of the things you might want to think about. Some of them we’ve covered on this podcast and Carla ‘s also got a very good webinar we did a few months ago on guerrilla testing apps, which you might want to check out. You can find the link on that. If you join our Slack group on UX Mentor. To me that’s free to join. Yeah. I don’t have too much else on it. Really. It is scary, but it’s a very good two way of kind of immersing yourself in user research, getting people in context and just getting used to talking to people and getting information out of them as well. I think that’s quite a useful skill in general.

23:16 Carla: Yeah, you can’t be shy. Do you have to just go for it? The first one will be always painful in any research you do. The first one is the worst cause that’s what you learn though. This thing doesn’t work with this cause discussion when it’s too long or maybe this wasn’t the right location. Is that you learn from the first one, that’s the most painful one. But the rest of them are, the following ones are okay. And people you actually start enjoying it. So, yeah. And also one thing before we go and then you can ask me to shut up. Is just think about how you can analyze as you go. So there are things like Affinity Mapping and you’ll see that in my webinar. Little tools to help you start identifying the common themes and the common patterns of your research, as you go, so that you don’t have to, uh, you know, then spend lots of time reading through your notes and remember when people did, a set. So it’s always as you say, like having a break after each one of the sessions and with your partner, with just someone you know, trying to write down the things that you learned, and (X) the kind of nicer quotes that you got from that particular bite piece of research and it just helps you then at the end of the day, you don’t have to go back to lots of recording, which is quite time consuming.

24:35 Chris: Yeah, I think we’ll probably do an episode on research analysis in general at some point as well. Okay. Is it plug time?

24:44 Carla: Yes, plug time. You love plug time. Go for it.

24:48 Chris: So I’ve already plugged the Slack channel, but I’ll do it again. We’ve got a community of around nearly 600 people now actually.

24:57 Carla: Oh Wow. That’s a lot of people.

24:59 Chris: That is a lot of people. So if you want to connect to have people learning UX and talk about the podcast or anything else you want to talk about. Then you can go to UXMentorMe and join for free there, in terms of design and go to we are on Twitter @DesignUntangled or on the web www.designuntangled.co.uk. Individually or on Twitter @Chris_Mears_ UX or @CarlaLindarte. I think that’s it. So this new little podcast recording tool we’re using lets me play the awesome intro and outro music at the click of a button. So should we give it a try and see if it works?

25:44 Carla: Adios Chris, I’ll see you in two weeks.

25:47 Chris: Okay. See you later.