Do telephone interviews undercount smokers?

Earlier this month, Statistics Canada released the most recent results from its long-standing and impressive Canadian Community Health Survey (CCHS). The results were very encouraging for those who work to reduce tobacco use -- smoking rates had declined to 13% -  the lowest rate in our lifetime. Earlier in the year, another Statistics Canada survey (the Canadian Tobacco and Nicotine Survey, CTNS) had estimated that smoking rates had fallen even further -- to 10%. 

This is not a trivial difference in measurement of this major health issue: one study finds more than a half million more smokers than another. Nor is it the first time that the CCHS provided higher estimates than other surveys conducted for Health Canada. 

For two decades there has been a gap of about 2 to 4 percentage points between the 'official' estimates produced by government surveys and used as indicators to track and assess programming and policy.

This post investigates whether different technologies used by these surveys contribute to the differences in these estimates, and also whether recent changes to CCHS methods may contribute to some of the decline.

Statistics Canada has reduced the number of in-person visits to collect information about the health of Canadians. 

Data for the smoking surveys sponsored by Health Canada, including the Canadian Tobacco Use Monitoring Survey (CTUMS), the Canadian Tobacco, Alcohol and Drugs Survey (CTADS) and the Canadian Tobacco and Nicotine Survey CTNS) are collected through telephone interviews conducted from Statistics Canada's call centre based in Ottawa. 

This call centre is also used to conduct the majority of the interviews for the Canadian Community Health Survey. This survey also has a parallel data collection process conducted by Statistics Canada's field interviewers, who are based across the country. These interviewers are trained to make personal contact with the dwelling that has been randomly selected and to conduct the interview in person (often at a at respondent's home). If that is not possible, they then attempt to make contact by telephone. 

During the revamping of the CCHS in 2015, the number of interviews conducted by regional staff was reduced, from about 50% to 30%. During the same period, the share of these interviews that were conducted in person also fell - from about three-quarters in 2007-2008 to less than two-thirds a decade later. Statistics Canada recently reported that all in-person interviews were terminated after March 2020 and no information is yet available for 2019.

The diminishing share of person interviews to CCHS results for 2005 to 2017-2018 is  shown below. 

More people will say they smoke when they are speaking in person to a surveyor

Compared with telephone interviews, in person discussions appear more likely to result in Canadians identifying themselves as smokers. 

In 2017 and 2018, for example, across all ages and both sexes those who were asked "At the present time, do you smoke cigarettes every day, occasionally or not at all?" were more likely to say they smoked if the interviewer could see them than if they were being interviewed on the telephone.

The difference was substantial: in 2017-2018 smoking prevalence estimates resulting from face-to-face discussions for men and women were about 50% higher (27% vs 18% for men and 19% vs 12% for women).

This survey mode effect appears to have grown over time

The gap between smoking rates estimated by in person interviews and telephone interviews has appeared in all of the CCHS waves over the past 15 years, but appears to have grown in recent years. 

In 2005, there was a 1 percentage point gap between the estimate of smoking prevalence produced by 47,500 in person interviews and those produced by 84,800 phone interviews (22.4% vs 21.4%). A decade later , in 2015-2016, the difference had grown to 9 percentage points (25% estimated 19,600 in person interviews vs. 16.1% estimated by 90,000 telephone interviews). 

Notably, the number of interviews conducted in person for CCHS (19,600 in 2017-2018) is greater than the number of interviews conducted for the CTNS (8,614 in 2019) or CTADS (16,349 in 2017). 

The impact of survey mode on smoking prevalence estimates has been identified before.

Early in the development of the CCHS, Statistics Canada tested for differences in responses received from their regional surveyors (CAPI) and those received by their call centres (CATI). They did not look specifically at the 60% of CAPI interviews that were conducted in person, but nonetheless found large and statistically significant different estimates of smoking prevalence of people in their 20s  (38% vs. 28%). They also found more Canadians self-reported other health conditions to regional staff, including obesity, drunk driving, and visiting a doctor.

Similar differences have been noted by American researchers, where estimates of smoking prevalence resulting from in person surveys were 3 percentage points higher than telephone surveys, and where it was found that some people are less likely to admit that smoking is permitted in their home when they are being interviewed face-to-face.

However, other studies have validated that self-reported smoking prevalence from CCHS is consistent with blood cotinine levels collected as part of the  Canadian Health Measures Survey (CHMS). This study was conducted with data collected in 2012-2013.

Statistics Canada has put other disclaimers on the 2020 CCHS results.

The diminished contribution of in-person interviews is only one reason why the 2020 estimates for smoking prevalence may be downwardly biased.

Last week, when releasing the 2020 results, Statistics Canada also signalled concerns to readers about  a higher non-response rate and a shortened collection period. "Users are advised to use the CCHS 2020 data with caution, especially when creating estimates for small sub-populations or when comparing to other CCHS years."

In particular, they noted that population groups which generally have higher smoking rates were under-represented in this survey period: "respondents interviewed during the pandemic (September to December 2020) appear to have slightly higher educational attainment and they were slightly more likely to own their place of residence compared to previous cycles." 

Implications for public health:

* Canadians should be cautious in interpreting the 2020 CCHS results for smoking behaviour as an indication that smoking has declined significantly during the COVID crisis.

* A review of the impact of survey modes on estimates of smoking behaviour is warranted.

A data sheet with CCHS estimates of smoking behaviour produced by telephone and in person interviews between 2005 and 2017-2018 is downloadable here.