Evaluating international migration products to strengthen and inform program scaling

Evaluative Research, Program Evaluation, User Experience, Migration, Emergency Response, Humanitarian Affairs

Introduction

The International Organization for Migration (IOM) is the United Nations agency promoting safe, humane, and orderly migration for the socioeconomic benefit of all. IOM works to manage migration, nurture international cooperation, find solutions to migration challenges, and provide humanitarian assistance to migrants in need. IOM advocates for the fundamental human right to freedom of movement, and educates on the fundamental links between migration and human development.

From 2017 to 2018, I had the privilege of working with IOM with 6 other consultants and researchers to: Madeline Achurch, Cora Griffin, Kruti Patel, Mary Pham, Stephanie Ullrich, and myself, along with our advisor, Kevin Hong. Together we collaborated to gauge the impact of a program pilot and identify opportunities for optimization and further development.

The Challenge

One of IOM’s programs, the Migrants in Countries in Crisis (MICIC) Initiative, was established in 2014 to assist migrants affected by conflict or natural disaster in a country outside of their origin location. It did so through a range of products and services: a comprehensive guidebook for crisis management with specific care for migrant populations, training initiatives for national migration authorities and emergency response authorities, digital e-training workshops, and digital apps with information for migrants. These products were targeted to users at multiple levels including migrants themselves, migrant assistance workers, NGOs and NPOs, and public authorities. The Initiative was piloted in Guatemala, Mexico, Myanmar, and Thailand, with plans to scale to further countries.

IOM commissioned our team to conduct an evaluation of the MICIC Program across the four pilot countries. Our goals were to provide an independent perspective of the MICIC Program’s effectiveness and utility for its users, identify stakeholders’ challenges and pain points to inform current and future programming, and provide actionable recommendations to strengthen and scale the MICIC Initiative.

Approach & Process

Theory of Change

The theory of change is a standard tool used in the social impact space to help organizations map their activities to their intended social outcomes and impacts. It is helpful in articulating the organization’s assumptions of how their products will impact users, and thus lays a framework to research and test those hypotheses regarding user experience and behavior. It’s not necessarily a UX tool, but was highly relevant for the purposes of this research project.

The MICIC Program consisted of many complex components, each with a diverse set of users and stakeholders, and unique characteristics in each country. Users themselves were also complex, and their touch-points with MICIC services interspersed depending on varying purposes and motivations. We first set out to understand and concretely define both the program’s products and its goals holistically. This involved consulting the IOM country teams, documents, and reviewing the products themselves.

The theory of change would later enable us to compare intended outcomes against results evidenced among users. It would also provide some organizational knowledge for service mapping activities directly with users — the two methods synergizing nicely to reveal how designing services and products on data-driven user insights could lead to the social impact that IOM sought to achieve.

Interview + Focus Group

Our team conducted primary research across all four countries—I conducted research in Mexico and Guatemala with two other teammates, while three other researchers traveled to Myanmar and Thailand. We had mapped the different stakeholders beforehand and worked with country teams to schedule logistics and operations. Interviews allowed for a deep dive of key concepts and for exploring nuanced sentiments and interactions between stakeholders and the MICIC services. Specific protocols and interview questions were prepared in advance to assess our variables of interest and tailored to different stakeholders and users. Interviews were recorded, and in many cases we had to rely on an interpreter (simultaneously in Latin America and consecutively in Asia). In some cases, we interviewed small groups of individuals to gather collective thoughts and observe interactions between group members. We transcribed all of our notes as soon as possible and wrote down impressions, trends, observations, new questions after each interview.

Service Mapping

For some individuals, we undertook a service blueprinting exercise to better understand their touch-points with the MICIC program as a part of the interview process. The purpose was to map the their experience with the MICIC services provided, and flesh out the coordination between internal actors, the direction of relationships, roles, and responsibilities in relation to emergency preparedness, response, and recovery for migrants in crises. We asked our respondents to draw out these interaction points with the range of actors involved in the MICIC program, talking them through their motivations, processes, and overall experience in those interactions.

As the points of interaction were often complex and difficult to articulate, this method enabled us to ground our conversation in a visual diagram. The blueprints themselves were a huge supplement to our interview data, visually communicating processes and helping us achieve more nuanced understandings of challenges and opportunities for optimizing services.

Observation + Contextual Inquiry

While we were in Guatemala, Mexico, Myanmar, and Thailand, we visited multiple offices of emergency response organizations, migrant shelters, and attended a MICIC training workshop to observe stakeholders throughout the program activities. The different environments afforded us the opportunity to observe program participants in their context of work, getting an overview of their activities and their usage of the MICIC products.

In addition to the above research methods, we also looked into public datasets, website analytics, log data from the MICIC digital apps, and reviewed and evaluated the MICIC guidebook materials.

Synthesis & Impact

We ended up gathering a somewhat intimidating amount of raw data in various forms—interview transcripts, visual diagrams, blueprints, audio recordings, datasets, observation notes, etc. We processed and coded all of our notes with NVivo, which allowed us to then explore and make sense of what people were saying regarding various themes in a much more systematic manner. We held multiple sessions to synthesize all that we were learning about outcomes of the MICIC program, the different situations and experiences of stakeholders, challenges that users faced in using MICIC products. We looked at the data through numerous lenses, looking for trends and patterns in certain countries, across all countries, with certain stakeholders, and across all stakeholders. We then translated those insights into strategic recommendations, and packaged our research into various communication platforms.

This project was one of my most intense but rewarding research experiences, an awesome opportunity to see exactly how we can make strides for social good by knowing our users and designing better experiences for them. If you’re interested in learning more about the process, the insights, our recommendations, challenges, the next steps, or my personal experience, then let’s chat!

Many thanks to my brilliant research partners: Madeline Achurch, Cora Griffin, Kruti Patel, Mary Pham, Stephanie Ullrich, and our advisor Kevin Hong. Thank you as well to the IOM country teams who graciously hosted us, and to all the individuals who entrusted us with their valuable time and stories.

“[Tim] was wonderful to work with, I'm so grateful to have had such a committed and reliable person on my team who truly cared for the project, the people we spoke to, and his teammates.” —Madeline Achurch, Project Manager

“[Tim] has an elevated self-awareness that I believe will serve his future very well. He was always professional and diplomatic, even in charged settings. His skills in design are impeccable and he impressed me with his ability to absorb and process a lot of information quickly.” —Stephanie Ullrich

See more photos from my time in Mexico and Guatemala.