Sometimes, people shy away from qualitative methods, thinking that they are only descriptive and don’t give you “hard” data. In fact, you can use qualitative methods to gather deeper insights, as well as to derive quantitative outcomes. All you need is the right questions and the right coding techniques.
In this section we look at some key qualitative methods:
Observation
Interviews and focus groups, including how to code results
Arts-based evaluation
Observation
Play Me, I’m Yours, Artist: Luke Jerram, Arts Centre Melbourne 2014. Source: Bailey & Yang Consultants (2014) Play Me, I’m Yours Project Evaluation (Betty Amsden Participation Program), Arts Centre Melbourne, Melbourne
Observation: Pros and Cons
Observation: Steps
Observation Form: Example
Interviews and Focus Groups
Interview styles can range from open-ended through to quite formally structured. We tend to conduct semi-structured interviews, using a discussion guide to make sure we cover key areas but allowing us flexibility to follow the interviewee’s line of thought. We also make sure there are a handful of key questions we ask all the interviewees to ensure we have some consistency.
With some groups e.g young people, I prefer to conduct interviews with “friendship triads” which just means groups of three people who know each other. This seems to help people bounce off each other. But one-on-one interviews are fine.
Focus groups are useful when you have to talk to a lot of people, but you do need to try to avoid group-think. We tend to limit online focus groups to 4-5 people, and in-person groups to 5-8 at the most.
Walking interviews are a useful approach when you are interested in people’s relationship to a particular place. They are also great if you want to hand the “power” back to the interviewee, so you are not in the position of expert any more, but in a position of not-knowing and learning from the interviewee.
Interview Process
The following is a rough process when using interviews to evaluate programs:
Stage 1: Design the interview guide
Stage 2: Conduct the interview
Stage 3: transcribe the interview
Stage 4: Code the interview
Stage 5: Analyse the codes
Stage 6: Report, and add illustrative quotes for key thematic findings
We ask permission to record interviews and focus groups so we can transcribe and code later. This is good to do, because it helps us avoid simply summarising our own interpretations of what people have said.
In the pre-amble to an interview, we always emphasise a few key points:
We ask permission to record
We explain that we are just here to listen and understand, and that there are no right or wrong answers
We ask for their frank and honest feedback, because that is the only way to improve a program
Sometimes we ask an interviewee to reflect on an experience and tell us about moments that have stayed with them. It can be useful to flag you are going to ask this “e.g. you don’t have to answer this now, but later it would be great to hear about a moment that has stayed with you, if any…”. This allows the question to simmer away in the back of the interviewee’s head so they aren’t taken by surprise.
It can also be a good idea to ask your most “important” questions towards the end of the interview. By then you have hopefully established some rapport with the interviewee.
Throughout the interview, be mindful of your body language, the noises you make, your facial expression - anything which might unconsciously bias the interviewee to try to tell you what you want to hear. A lot of communication is non-verbal. This doesn’t men you have to be deadpan - but whatever they say, you have to be ready to be sympathetic and encouraging. Your job is not to get them to answer questions in a particular way, but to let them reflect on an experience. You are like a journal, with prompts.
Coding
Coding is the thing that seems to scare people so I will run through it here. Basically, coding means reviewing the interview transcript and identifying the key themes that the person has talked about.
You’ll start out with fairly verbatim themes, then gradually, as you do more transcripts, common themes will start to emerge and you can start grouping them.
It’s a cumulative process: you have to start out allowing for a fair bit of detail in a theme, and then gradually refining it.
The way I do it: I print out the transcript and jot down on the left-hand side the category of enquiry the person seems to be talking about, and on the right-hand side I jot down the theme of what the person is saying. It looks something like the below.
What my first stage of “coding” looks like.
The “themes” become the initial codes, and the “categories” become the areas of enquiry which I gather the themes against.
From here, I start to transcribe my themes into a spreadsheet which will end up being a very wide spreadsheet - the below images are examples of parts of the spreadsheet. Because a person might say more than one thing which relates to a category, you have to include multiple columns for each category.
I type in the “code”, e.g. under personal goals I might include “working with artists from overseas”. I do this for each thing the participant said. Gradually I refine the codes so they might start to cohere, e.g “work with artist from overseas” could eventually be grouped under “artistic development”.
Categories may correspond to some of the main areas of questions in the interview. However, don’t assume this is the case: there might be some unexpected comments which it is important to include and categorise.
The categories may also match up with the areas I identified as part of my impact framework. But they won’t always match up, which is why it is important to code the interviews as they are, and not attempt to code them using your impact framework language in the first instance. You have to be as objective as possible, and not subconsciously shoe-horn the interviewee’s comments into what you thought the impact would be.
Templates
Here are some templates for interview discussion guides, note templates and coding templates.
Interview Summary Template (we use this as an interim step if one person did the interview and then another person is doing the coding. You can also use this if there aren’t very many interviews so you don’t need the massive spreadsheet)
Walking Interviews
Walking Interviews were used in the “Connected Lives Project” (UK). These are examples of the routes and photos taken on two walking interviews. Source: Clark, A. Emmel, N. (2010) “Realities Toolkit #3 Using walking interviews.” Realities, part of the ESRC National Centre for Research Methods, Manchester University, UK.
Walking interviews are interviews which you do whilst walking. They are useful when you want to understand people’s relationship with a particular place. They are also useful when you want to hand the “power” back to the interviewee, so you are not in the role of “expert” - the interviewee chooses the route and may talk more comfortably in motion.
Walking Interviews: Pros and Cons
Walking Interviews: Steps
Arts-Based Evaluation
When you are working in the arts, it seems to makes sense to use creative tools to evaluate the impact of a program. However, it can be tricky to interpret the artefacts which participants create.
Arts-Based Evaluation: Pros and Cons
Arts-Based Evaluation: Steps
Arts-Based Evaluation: Examples
Youthrex Webinar, 2015
Art Jam
Art Jams are a gathering of individuals who make art by way of collaboration, improvisation and responsiveness. Art Jams are like focus groups, but with meaning arising about an evaluation topic through the interactive mode of art-making, rather than conversation e.g. facilitators can document the jam to collect important data about the experience.
transformational self-portraits
Participants create a series of self-portraits at different intervals of a project (painting, drawing, collage, or any other artform).
Stage 1. depicts how participants see themselves, things that have influenced their sense of self and their motivation for participating in the project.
Stage 2. participants represent themselves in what they see as their role or contribution to the project.
Stage 3. participants create a portrait of how they have changed as a result of the project and /or how they imagine taking what they have learned to shape their future selves.
Before and After Tool
We use this quite a lot, especially with kids and young people. We ask respondents to reflect on how they were before the activity and afterwards. Then we code the responses according to emotional valence and arousal, to see if there has been an overall change in mood amongst participants.
Use this Before and After Coding Spreadsheet Template to code responses. You can also use this type of spreadsheet to code the transformational self-portraits.
Sources
Better Evaluation, Collect and/or Retrieve Data, Better Evaluation.
Burns, L. Frost, J. (2010) Arts informed evaluation: A Creative Approach to Assessing Community Arts Practices, Arts for Children and Youth Toronto, VIBE Arts, Canada.
Clark, A. Emmel, N. (2010) “Realities Toolkit #3 Using walking interviews.” Realities, part of the ESRC National Centre for Research Methods, Manchester University, UK.
Daykin, N.(2015) ”Creative and Arts Based Evaluation Methods.” Credible and Credible.
Kinney, P. (2017). “Walking Interviews” Social Research Update. Issue 67, Summer. University of Surrey.
Macfarlane, A. (2017) “Non-Participant Observation” Better Evaluation, Sharing information to improve evaluation, the Australia and New Zealand School of Government (ANZSOG), Melbourne.
Searle, M. (2016) “Capturing the Imagination: Arts-Informed Inquiry as a Method in Program Evaluation,“ Canadian Journal of Program Evaluation Spring: 34-60.
Spicksley, K. (2018) “Walking interviews: A participatory research tool with legs?” The Bera Blog. Research Matters British Educational Research Association.
Van der Vaart, G. van Hoven, B. and P.P. Huigen, P. (2018) “Creative and Arts-Based Research Methods in Academic Research. Lessons from a Participatory Research Project in the Netherlands” FQS Forum: Qualitative Social Research Volume 19, No. 2, Art. 19 – May.
Wang, Q. Coemans, S. Siegesmund, R. Hannes, K.(2017) ”Arts-based methods in Socially Engaged Research Practice: a Classification Framework”. Art/Research International: A Transdisciplinary Journal. 2017;2(2): 5–39.
Wimpenny, K. and Savin-Baden. M. “Using Theatre and Performance for Promoting Health and Wellbeing Amongst the 50+ Community: An Arts-Informed Evaluation,” The International Journal of Social, Political and Community Agendas in the Arts, 8(1): 47-64.