Unfounded: The methodology behind The Globe’s survey of Canadian police services

In the 10 months since The Globe and Mail launched its investigation into how Canadian police handle sexual assault, dozens of services from across the country have pledged to go back and review previously closed files – in particular, cases closed as "unfounded," a term that means the investigating officer does not believe a crime occurred.

But, on first glance, the nature, scope and goals of those reviews seemed to vary dramatically. Further, because not all services publicly announced their plans, there was no way to figure out exactly how many departments were conducting audits. In order to get a better picture of sexual assault case reform, The Globe developed an 18-question survey for 177 police services across the country.

Using this data, The Globe has created a look-up tool that builds off of the "Will They Believe You" search feature of the original Unfounded series, which enabled anyone in Canada to look up what their given police service was doing. The information included in the first phase of Unfounded is below each police service's survey results.

The survey included questions about the review itself. Was a service doing one? If so, what types of cases were being looked at and by whom? How many files were being reviewed? If the review was finished, how many cases had been reclassified or reopened? We also asked whether any policy or training changes were made as a result of its findings.

View the complete survey here.

We tried to keep the questions to a multiple-choice format, so that we would be able to easily compare services. We also provided space, when appropriate, for "other" options. The final question on the survey was an open-ended one: "Is there anything else that you would like to add to help us better understand your police service's approach to handling sexual assault cases?" We have included the service's full answer to this question in each profile. Each profile also contains a "Notes" section, where The Globe kept a record of unique issues that arose pertaining to that service. For example, when relevant we offered a survey in both English and French. For police services that replied in French, we noted that it has been translated. In this section, we also included any additional comments made by the given police service.

The 177 police services identified for the survey came from a list of what is known as "police respondent populations" (PRPs), which we obtained from the Canadian Centre for Justice Statistics, a division of Statistics Canada. The version of this list from 2014 served as the basis for our initial Unfounded investigation. We sent the survey to the services on the 2015 list, which was the most up-to-date at the time. There are more than 1,100 police jurisdictions in Canada. Each municipal police service is responsible for one. The remainders are divided between the RCMP, the Ontario Provincial Police, the Sûreté du Québec and the Royal Newfoundland Constabulary. We sent only one survey to each of the aforementioned police services and they responded from the perspective of the organization as a whole rather than hundreds of times relating to each of their jurisdictions. In four instances, it appeared municipal services on this list had been absorbed by either the RCMP or SQ.

We sent the surveys out in May, 2017. The survey was automated and each police service was provided with a unique user login and password. Some police services had trouble accessing the document, possibly due to firewalls on their networks. In these cases, we emailed the service a PDF version. Some replied using the PDF, others responded in a word document and others in the body of an email. In these cases, The Globe transferred these answers into the automated survey. Whenever a police service did not physically type out their own answers, a Globe proofreader verified the completed survey against the original.

While some police services responded right away, others said they would not be able to reply until later in their review process. Others did not acknowledge The Globe's request. Over the next four months, we circled back to each non-responsive police service at a minimum of three times to ensure that the service was aware of the survey.

Ultimately, about half of the country's services responded to our survey. However, in some instances, by tracking police reviews through their respective local media, The Globe was able to complete at least some information on the document. In these instances, we used the media reports conservatively and only pulled the most basic information. We have noted each profile where information was collected in this way. In other cases, police services that completed reviews prepared a report on their findings for their local police services board. The Globe was able to obtain some of these reports and, again, complete some parts of the survey. These profiles are also noted. Finally, in the course of circling back to each police service to check on the status of a survey, some police services offered to speak with The Globe via phone about the survey content. In other cases, a police service declined to complete the document, but offered a formal interview. In the course of that interview, The Globe asked questions that were also posed by the survey. These profiles are also noted appropriately.

Generally, The Globe has not fact-checked police services' answers. If a service indicated that it has implemented "trauma-informed training" we have taken them at their word. That said, for numerous reasons, The Globe had to reach out to dozens of police services with additional questions or clarifications relating to the answers provided. The most common reason we would investigate an answer – not taking it at face value – was because of something written in the "notes" or "other" field. This "survey cleaning" was necessary in order for us to analyze the data.

For example, sometimes there appeared to be typos or obvious errors. In other cases, The Globe required clarification for a logic issue. For example, sometimes a police service indicated that, "yes," it would be doing a case review, but also answered a question about why it had chosen not to review cases. In other instances, a service would say it had included violence against women (VAW) advocates in its review, but then in the notes section below, would further explain that the first phase of a review was actually done internally and VAW advocates would be involved going forward. And in some cases, discussions about one question would lead to a discussion about another related question and a service indicated that a different answer would be more appropriate.

Occasionally, a police service would have a different interpretation of a question than we intended. This seemed to be a problem for about 10 police services in a question we posed about future reviews. In reporting on the sexual assault case reviews that are taking place across the country, The Globe has drawn a distinction between supervisor oversight of current cases for quality, as opposed to historical reviews, in which previously closed cases are revisited and reexamined. We asked police services that indicated they would be conducting future reviews if those audits would be done annually, and, if not, at what frequency. In some cases, police services said their cases would be reviewed "daily." In other cases, the service wrote that supervisors always review cases before they are closed. These types of answers prompted clarifying phone calls and in most cases the police service agreed their original answer was not appropriate. Sometimes, however, they believed their answer was correct, despite the fact it did not meet The Globe's definition. In these cases, The Globe left their answer as it was reported, but excluded it from our national analysis. Whenever The Globe sought clarification or made any change, the profile is noted.

Another key issue relating to The Globe's survey pertains to a question about the types of cases reviewed. We initially asked police services to check one from a multiple choice list that included options such as "all unfounded cases" and "a sample of sexual assault" cases. We also offered an "other" field. A significant number of services either selected both "unfounded" and "a sample of sexual assault" files or wrote that information in the "other" field. As such, we made the decision to add an option of "all founded cases as well as a sample of sexual assault" cases in our analysis. Also, many police services noted in the "other" field that they were reviewing all sexual assault cases, so we also added that as an option. Similarly, many police services indicated that they were considering or planning a VAW review, but our original survey did not specifically provide that as an option. For ease of comparison, we added that indicator. Finally, our initial survey only included the year options going back a decade, but a handful of police services checked every box, which begged the question: Did they go back further than 2007? After seeking clarification, we learned that yes, in some instances police services did go back further. For this reason we added 2005 and 2006. We contacted every police service that filled out to 2007 to see if their review went further.

By the end of September, the majority of police services had been contacted at least three times, so The Globe began cleaning the surveys. Later, because of the extent of the changes, we had a fact-checker go through each survey to compare the original responses and track changes against the completed survey. Globe researchers also fact-checked the surveys that had been partially filled out through media reports and board reports.

Before publication, The Globe emailed all 177 police services a copy of the survey to see if they wished to make any changes or if they spotted any errors. Dozens made changes. A number of police services that initially did not participate decided to complete the survey. The Globe accommodated each request but after two weeks had to set a firm cut-off date for changes. At this point, the survey was sent out again for a final viewing. Only overt errors could be changed.

The Globe conducted the analysis using statistical software that allowed us to query the data in repeatable and reproducable form. As with the original Unfounded series, our analysis was verified by a data scientist who used a parallel process and reached the same results for each of our data queries.