UNHCR used phone surveys in Mozambique recently to gather information and data on living conditions of forcibly displaced people in the country. © UNHCR
By Ilgi Bozdag and Hyunju Park
Sampling hard-to-reach populations has been a challenge for surveys for a long time. In the context of forced displacement, out-of-camp refugees, and internally displaced persons (IDPs) are particularly difficult to include in a sampling frame due to higher mobility compared to populations who reside in camps. UNHCR, the UN Refugee Agency, estimates that around 60 percent of refugees and 80 percent of IDPs live in urban environments. With global forced displacement reaching 108 million at the end of 2022, it is more important than ever to have high-quality data on the living conditions of people forced to flee.
Case study in Mozambique
In 2021, UNHCR launched its Results Monitoring Surveys (RMS), which are household-level surveys used to facilitate and harmonize the monitoring of changes in the lives of forcibly displaced and stateless persons. Mozambique is one of more than 30 countries where the RMS have already been implemented. The country hosts around 5,200 refugees and 26,900 asylum-seekers in May 2023.[1] Among those, around 9,200 live in the Maratane Camp, which is the only refugee settlement in the Nampula province. The RMS provided the opportunity to understand the situation of populations residing both inside the Maratane settlement via Computer-Assisted in-Person Interviews (CAPI) as well as outside the Maratane settlement via Computer-Assisted Telephone Interview (CATI)-Assisted Telephone Interviews.
Computer-Assisted Telephone Interviews for forcibly displaced people living outside of camps
ProGres, UNHCR’s registration and case management tool, enabled a randomly selected sample covering refugees, asylum-seekers and others of concern. Populations residing in the Maratane settlement and Nampula City were excluded, as they were interviewed face-to-face (CAPI).
The CATI sampling frame was made up of 15,792 individuals who were registered with phone numbers (4,804 women and 10,988 men). As phone ownership is biased towards men, stratification was applied to ensure a 40 percent female representation within the survey despite them representing only 30 percent of the sampling frame. The target sample size for both the CATI and CAPI was 400, i.e. 800 refugees and asylum-seekers in Mozambique in total.
On the phone, the standard RMS questionnaire takes around 30 minutes to be answered, and the survey was carried out in Portuguese. For the CATI survey, enumerators were instructed to make ten attempts to reach each selected individual. From the analysis, however, four attempts appeared to be enough. After the fourth attempt, it was very unlikely to have an answer as the chart on the right demonstrates.
Sampling is key for successful phone surveys
The case study from Mozambique provided some insights on advantages and caveats of remote data collection of forcibly displaced. In a forced displacement setting, random dialing methods (i.e. covering the whole population in a sample) are impossible to apply since refugees, asylum-seekers and stateless people constitute only a subset of the population.
A major verification exercise for refugees, asylum-seekers, and others of concern in the proGres database of Mozambique was conducted in parallel with the phone survey. Hereby, UNHCR verifies and updates the information of these populations in the country. This enabled UNHCR statisticians to observe the differences before and after the clean-up of the database, as ProGres was the source of sampling frame for the CATI. In the first batch of samples (pre-verification), 150 interviews were completed out of 1,730 attempted calls, while 265 interviews out of 1,148 attempts were successfully done for the second batch (post-verification). The probability of reaching someone decreases faster in the second batch as the number of attempts grows, since the person not reached is unlikely to have an active phone number in the updated database.
In addition to the number of call attempts, the phone survey in Mozambique showed that sampling is better done at the individual rather than at the household level. This enables the capturing of unbiased information for questions aimed at individuals.
Results show stark differences
The phone survey captured the socioeconomic situation of refugees and asylum-seekers in Mozambique, and some results present stark contrasts for those in the Maratane camp compared to those living outside a camp. For example, those residing in the settlement are more likely to be covered by social protection systems than those living in non-settlements. Such differences were not clearly observed before, and the remote data collection allowed it.
CATI offers great potential to enhance understanding of the living conditions of hard-to-reach populations, especially in forced displacement context. The methodologies have room for improvement, such as the inherent bias of phone surveys as well as the variety of languages in which refugees and asylum-seekers are comfortable to answer. Going forward, UNHCR will continue to explore and invest in innovative ways to improve data quality, while addressing the limitations.
[1] In addition, there are 600 ‘Others of concern to UNHCR’ which include people who do not fall into other groups of forced displacement but still need protection and/or assistance. In Mozambique, these are usually family members, that is, a spouse or children, of refugees or asylum-seekers. See the definition of population groups reported by UNHCR.