Conceptual Framework
A great deal of scholarship has been devoted to the study of religious freedom and the extent of restrictions on religion. Much of this research relies upon case studies, assessments by observers within countries and reviews of news reports. This research has yielded valuable insights and has helped to call attention to places and circumstances in which religious practices or beliefs are infringed on by governments or societies. A more systematic assessment and comparison of restrictions on religion worldwide, however, requires the development of valid and reliable quantitative indicators.
Good measurement entails the translation of abstract concepts (in this case, “restrictions on religion”) into factual indicators. This translation requires indicators that satisfy several criteria. First, they must be comprehensive, covering a broad range of facets of the issue, since no single indicator, or even small set of indicators, could be expected to capture all the ways in which religion might be restricted by government or in society. Moreover, individual indicators can be affected by one-time events or temporary circumstances. The use of multiple indicators, therefore, helps to ensure that a wide range of important manifestations of restrictions on religion are captured, and also helps to minimize the impact of any single indicator on the overall score.
For the index of government restrictions on religion, creating a comprehensive measure began with the identification by the Pew Forum’s research team of four main ways in which such restrictions occur: (1) constitutional restrictions or restrictions based in national law or policy; (2) restrictions imposed by government officials at any level, whether codified in law or not; (3) use of force or coercion against religious groups by government agencies or their representatives; and (4) government favoritism toward particular religious groups. In each of these four areas, the research team developed multiple indicators, such as determining whether a country’s constitution specifically provides for “freedom of religion,” or whether it establishes a favored religion or religions. A total of 20 separate indicators make up the Government Restrictions Index.
For the measurement of social hostilities involving religion, the Pew Forum’s researchers identified three principal ways in which social hostility toward religious groups is expressed: (1) crimes or malicious acts motivated by religious hatred or bias; (2) public religious tensions that lead to violence; and (3) religion-related terrorism and war. In each of these areas, multiple indicators were devised to capture a wide range of hostilities, from individual malicious acts to mob violence and nationwide armed conflict. A total of 13 indicators make up the Social Hostilities Index.
Second, accurate measurement requires that the multiple indicators used within each of the two indexes be internally consistent. Though the indicators may focus on widely varying kinds of restrictions on religion, all of them should work in tandem to identify meaningful levels of restrictions. Put differently, countries with high levels of restrictions on religion will typically, though not always, score higher on a given indicator than countries with lower levels of restrictions. If an indicator does not follow this pattern, then it may be measuring something other than the concept of restrictions on religion.
Third, good measures also are reliable. One aspect of reliability is the extent to which different observers attempting to apply the set of indicators will get the same result. If two researchers look at the same data sources and reach different conclusions about how a country should be scored on a particular indicator, then the measure lacks reliability. Another aspect is the extent to which the score on an indicator is consistent over time, assuming that the restriction itself has not changed during that period. If a nation’s constitution and laws have not changed from one year to the next, a reliable indicator of constitutional and legal restrictions on religion will yield the same result in both years.
Finally, measures must be valid. Validity refers to the extent to which the measure captures the abstract concept under examination – in this case, restrictions on religious beliefs and practices. One way of assessing validity is to compare the results of the index with the views of experts. For example, are countries that score very high on the Government Restrictions Index considered by experts in the field to be the most restrictive nations? Conversely, do experts believe that certain countries have a high level of restrictions even though the index indicates that the level is low? Another method of assessment is to compare scores on the index with other quantitative indicators of restrictions that appear to measure restrictions on religion but are not themselves included in the index. As discussed below, the indexes correspond closely with expert assessments of countries, and they correlate strongly with other indicators of government restrictions on religion and social hostilities involving religion.
Overview of Procedures
The methodology used by the Pew Forum to assess and compare restrictions on religion was developed by Senior Researcher Brian J. Grim in consultation with other members of the Pew Research Center staff, building on a methodology that Grim and Prof. Roger Finke developed while at Penn State University’s Association of Religion Data Archives.1 The goal was to devise quantifiable, objective and transparent measures of the extent to which governments and societal groups impinge on the practice of religion. The findings were used to rate 198 countries and self-governing territories on two indexes that are reproducible and can be periodically updated.
This research goes beyond previous efforts to assess restrictions on religion in several ways. First, the Pew Forum coded (categorized and counted) data from 16 published sources, providing a high degree of confidence in the findings. The Pew Forum’s coders looked to the sources only for specific, well-documented facts, not for opinions or commentary.
Second, the Pew Forum’s staff used extensive data-verification checks that reflect generally accepted best practices for such studies, such as double-blind coding (coders do not see each other’s ratings), inter-rater reliability assessments (checking for consistency among coders) and carefully monitored protocols to reconcile discrepancies between coders.
Third, the Pew Forum’s coding took into account whether the perpetrators of religion-related violence were governmental or private actors. The coding also identified how widespread and intensive the restrictions were in each country.
Fourth, two independently coded years of data (July 1, 2006, through June 30, 2007, and July 1, 2007, through June 30, 2008) were averaged to create solid baseline measures that are less affected by methodological or informational variability in any one year.
The indexes can be used to compare nations, groups of nations or regions of the world. But one of the most valuable uses of the indexes will not be realized immediately. That is the ability of the indexes to chart change over time. Using the current two-year average for each nation as a baseline, future editions of the index could assess increases or decreases in government restrictions and social hostilities.
Countries and Territories
The Pew Forum study covers a total of 198 countries and territories. These include all 192 states that were members of the United Nations during the period under examination (mid-2006 to mid-2008), with the exception of North Korea, for which sufficiently precise and timely data was not available. In addition, the study includes seven territories: Hong Kong, Macau, Taiwan, the Palestinian territories, Kosovo, Western Sahara and Northern Cyprus. These are treated as separate entities, for various reasons, by some or all of the primary information sources for this study. The U.S. State Department, for example, reports separately on Northern Cyprus because it has been administered by Turkish Cypriot authorities since 1974.
Although the 198 countries and territories vary widely in size, population, wealth, ethnic diversity, religious makeup and form of government, the study does not attempt to adjust for such differences. Poor countries are not scored differently on the indexes than wealthy ones. Countries with diverse ethnic and religious populations are not “expected” to have more social hostilities than countries with more homogeneous populations. And democracies are not assessed more leniently or harshly than authoritarian regimes. However, several charts and related passages in this report focus on countries of roughly similar size – such as the world’s 25 and 50 most populous nations – and on broad geographic regions. Those comparisons may be more instructive than comparing very large, populous countries to much smaller ones.
Information Sources
The Pew Forum identified 16 widely available, frequently cited sources of information on government restrictions and social hostilities involving religion around the world. These sources, which are listed below, include reports from U.S. government agencies, several independent, nongovernmental organizations and a variety of European and United Nations bodies. Although most of these organizations are based in Western countries, many of them depend on local staff to collect information across the globe. As previously noted, the Pew Forum did not use the commentaries, opinions or normative judgments of the sources; the sources were combed only for factual information on specific policies and actions.
Primary Sources
| 1. Country constitutions |
| 2. U.S. State Department annual reports on International Religious Freedom |
| 3. U.S. Commission on International Religious Freedom annual reports |
| 4. U.N. Special Rapporteur on Freedom of Religion or Belief reports (Asma Jahangir) |
| 5. Human Rights First reports |
| 6. Hudson Institute publication: Religious Freedom in the World (Paul Marshall) |
| 7. Human Rights Watch topical reports |
| 8. International Crisis Group country reports |
| 9. United Kingdom Foreign & Commonwealth Office annual report on human rights |
| 10. Council of the European Union annual report on human rights |
| 11. Amnesty International reports |
| 12. European Network Against Racism Shadow Reports |
| 13. United Nations High Commissioner for Refugees reports |
| 14. U.S. State Department annual Country Reports on Terrorism |
| 15. Anti-Defamation League reports |
| 16. U.S. State Department Country Reports on Human Rights Practices |
| U.S. government reports with information on the situation in the United States |
| 17. Dept. of Justice Report on Enforcement of Laws Protecting Religious Freedom 2000-2006 |
| 18. Department of Justice “Religious Freedom in Focus” newsletters |
| 19. FBI Hate Crime Reports |
The Pew Forum’s staff developed a battery of questions similar to a survey questionnaire. Coders consulted the primary sources in order to answer the questions separately for each country. While the U.S. State Department’s annual reports on International Religious Freedom generally contained the most comprehensive information, the other sources provided additional factual detail that was used to settle ambiguities, resolve contradictions and help in the proper scoring of each question.
The questionnaire, or coding instrument, generated a set of numerical measures on restrictions in each country. It also made it possible to see how government restrictions intersect with broader social tensions and incidents of violence or intimidation by private actors. The coding instrument with the list of questions used for this report is shown in the Summary of Results.
The coding process required the coders to check all the sources for each country. Coders determined whether each source: provided information critical to assigning a score; had supporting information but did not result in new facts; or had no available information on that particular country. Multiple sources of information were available for all countries and self-administering territories with populations greater than 1 million. More than three-in-four of the countries and territories analyzed by the Pew Forum were multi-sourced; only small, predominantly island, countries had a single source, namely, the U.S. State Department reports.
Coding the United States presented a special problem since it is not included in the State Department’s annual reports on International Religious Freedom. Accordingly, the Pew Forum’s coders also looked at reports from the Department of Justice and the FBI on violations of religious freedom in the United States, in addition to consulting all of the primary sources, including reports by the United Nations, Human Rights Watch, the International Crisis Group and the U.K. Foreign & Commonwealth Office, many of which do contain data on the United States.
The Coding Process
The Pew Forum employed strict training and rigorous coding protocols to make its coding as objective and reproducible as possible. Coders worked directly under a senior researcher’s supervision, with additional direction and support provided by other Pew Forum researchers. The coders underwent an intensive training period that included a thorough overview of the research objectives, information sources and methodology.
Countries were double-blind coded by two coders, and the initial ratings were entered into SPSS data files. The coders began by filling out the coding instrument for each country using the information source that had the most comprehensive information, typically the U.S. State Department reports. The protocol for each coder was to answer every question on which information was available in the initial source. Once a coder had completed that process, he or she then turned to the other sources. As new information was found, this was also coded and the source duly noted. Whenever ambiguities or contradictions arose, the source providing the most detailed, clearly documented evidence was used.
After two coders had separately completed the coding instrument for a particular country, their scores were compared by a senior researcher. Areas of discrepancy were discussed at length with the coders and were reconciled in order to arrive at a single score on each question for each country.
Throughout this process, the coding instrument itself was continually monitored for possible defects. The questions were designed to be precise, comprehensive and objective so that, based on the same data and definitions, the coding could be reliably reproduced by others with the same results.
Pew Forum staff generally found few cases in which one source contradicted another. When contradictions did arise – such as when sources provided differing estimates of the number of people displaced due to religion-related violence – the source that cited the most specific documentation was used. The coders were instructed to disregard broad, unsubstantiated generalizations regarding abuses and to focus on reports that contained clear, precise documentation and factual detail, such as names, dates and places where incidents occurred.
Inter-rater reliability statistics were computed by comparing the coders’ independent, blind ratings. The Pew Forum took scores from one coder for the 198 countries and compared them with another coder’s scores for the same questions, computing the degree to which the scores matched. These measures were very high, with an average score of .8 or above on the key variables. Scores above .8 on a 0-to-1 scale are generally considered very good, and scores around .7 are generally acceptable. The Pew Forum’s overall inter-rater reliability average across all the variables coded was .86.
The data-verification procedures, however, went beyond the inter-rater reliability statistics. They also involved comparing the answers on the main measures for each country with other closely related questions in the dataset. This provided a practical way to test the internal reliability of the data.
Pew Forum staff also checked the reliability of the Pew Forum’s coded data by comparing them with similar, though more limited, religious restrictions datasets. In particular, published government and social regulation of religion index scores are available from the Association of Religion Data Archives (for three years of data) and the Hudson Institute (for one year of data), which makes them ideal measures for cross validation. The review process found very few significant discrepancies in the coded data; changes were made only if warranted by a further review of the primary sources.
Restriction of Religion Indexes
The study measures the extent to which governments as well as private actors (social groups, organizations and individuals) restrict religious beliefs and practices in countries around the world. To establish baseline measures for each country, the Pew Forum averaged at least two independently coded years of data (July 1, 2006, through June 30, 2007, and July 1, 2007, through June 30, 2008, in all cases, and going back in some cases to July 1, 2005, to more accurately assess recurring societal tensions).
The Government Restrictions Index is based on 20 indicators of ways that national and local governments restrict religion, including through coercion and force. The Social Hostilities Index is based on 13 indicators of ways in which private individuals and social groups infringe on religious beliefs and practices, including religiously biased crimes, mob violence and efforts to stop particular religious groups from growing or operating. The study also counted the number and types of documented incidents of religion-related violence, including terrorism and armed conflict.
Government Restrictions Index
Coding multiple indicators makes it possible to construct a Government Restrictions Index (GRI) of sufficient gradation to allow for meaningful cross-national comparisons. An additional advantage of using multiple indicators is that it helps mitigate the effects of measurement error in any one variable, providing greater confidence in the overall measure.
The Pew Forum coded 20 indicators of government restrictions on religion (see the Summary of Results). In two cases, these items represent an aggregation of several closely related questions: Measures of five types of physical abuses are combined into a single variable (Question No. 19); and seven questions measuring aspects of government favoritism are combined into an overall favoritism scale (Question No. 20 is a summary variable showing whether a country received the maximum score on any one or more of the seven questions). These 20 items were added together to create the GRI.
A test of whether the 20 items were statistically reliable as a single index produced a scale reliability coefficient of .931. Since coefficients of .7 or higher are generally considered acceptable, it was appropriate to combine these 20 items into a single index.
The GRI is a fine-grained measure created by adding the 20 items on a 0-to-10 metric, with 0 indicating very low government restrictions on religion and 10 indicating extremely high restrictions. This involved two general calculations. First, the 20 questions that form the GRI were standardized so that each variable had an identical maximum value of 1 point, while gradations among the answers allowed for partial points to be given for lesser degrees of the particular government restriction being measured. Second, the overall value of the index was proportionally adjusted so that it had a maximum value of 10 and a possible range of 0 to 10.
Social Hostilities Index
In addition to government restrictions, violence and intimidation in societies also can limit religious beliefs and practices. Accordingly, Pew Forum staff tracked more than a dozen indicators of social impediments on religion. Once again, coding multiple indicators made it possible to construct an index that shows gradations of severity or intensity and allows for comparisons between countries. The Summary of Results contains the 13 items used by Pew Forum staff to create the Social Hostilities Index (SHI).
As with the Government Restrictions Index, various types of violence and intimidation were combined. A test of whether these 13 items were statistically reliable as a single index produced a scale reliability coefficient of .919. Since coefficients of .7 or higher are generally considered acceptable, it was statistically appropriate to combine these items into a single index.
The SHI was constructed by adding together the 13 indicators based on a 0-to-10 metric, with 0 indicating very low social impediments to religious beliefs and practices and 10 indicating extremely high impediments. This involved two general calculations. First, the various questions that form the index were standardized so that each variable had an identical maximum value of 1 point, while gradations among the answers allowed for partial points to be given for lesser degrees of the particular hostilities being measured. Second, the indicators were added together and set to have a possible range of 0 to 10 by multiplying each variable by 10/13.
Levels of Restrictions
The Pew Forum categorized the levels of government restrictions and social hostilities by percentiles. Countries with scores in the top 5% on each index are categorized as “very high.” The next highest 15% of scores are categorized as “high,” and the following 20% are categorized as “moderate.” The bottom 60% of scores are categorized as “low.” Readers should note that since the indexes measure the accumulated impact and severity of restrictions, distinctions among the scores of the countries in the bottom 60% of scores are less significant than distinctions made at the upper end of the indexes, where differences in the number and severity of restrictions between countries are greater. This is evident by the fact that the range of difference between scores of countries in the entire bottom 60% (0-2.3 on the GRI and 0-1.8 on the SHI) is about the same as the range of differences between scores in just the top 5% (6.7-8.4 on the GRI and 6.8-9.4 on the SHI).
Religion-Related Terrorism and Armed Conflict
Terrorism and war can have huge direct and indirect effects on religious groups, destroying religious sites, displacing whole communities and inflaming sectarian passions. Accordingly, the Pew Forum tallied the number, location and consequences of religion-related terrorism and armed conflict around the world, as reported in the same primary sources used to document other forms of intimidation and violence. However, war and terrorism are sufficiently complex that it is not always possible to determine the degree to which they are religiously motivated or state sponsored. Out of an abundance of caution, this study does not include them in the Government Restrictions Index. They are factored instead into the index of social hostilities involving religion, which includes one question specifically about religion-related terrorism and one question specifically about religion-related war or armed conflict. In addition, other measures in both indexes are likely to pick up spillover effects of war and terrorism on the level of religious tensions in society. For example, hate crimes, mob violence and sectarian fighting that occur in the aftermath of a terrorist attack or in the context of a religion-related war would be counted in the Social Hostilities Index, and laws or policies that clearly discriminate against a particular religious group would be registered on the Government Restrictions Index.
For the purposes of this study, the term religion-related terrorism is defined as premeditated, politically motivated violence perpetrated against noncombatant targets by subnational groups or clandestine agents that have some identifiable religious ideology or religious motivation. Readers should note that it is the political character and motivation of the groups, not solely the type of violence, that is at issue here. For instance, a bombing would not be classified as religion-related terrorism if there was no clearly discernible religious ideology or bias behind it.
Religion-related war or armed conflict is defined as armed conflict (a conflict that involves sustained casualties over time or more than 1,000 battle deaths) in which religious rhetoric is commonly used to justify the use of force, or in which one or more of the combatants primarily identifies itself or the opposing side by religion.
Potential Biases
As noted in the report, the primary sources indicate that the North Korean government is among the most repressive in the world, including toward religion. Because of independent observers’ lack of regular access to North Korea, however, the sources are unable to provide the kind of specific, timely information that forms the basis of this report. Therefore, North Korea is not included on either index.
This raises two important issues concerning potential information bias in the sources. The first is whether other countries that limit outsiders’ access and that may seek to obscure or distort their record on religious restrictions were adequately covered by the sources. Countries with relatively limited access have multiple primary sources of information that the Pew Forum used for its coding. Each is also covered by other secondary quantitative datasets on religious restrictions that have used a similar coding scheme, including earlier years of coded data from U.S. State Department reports previously produced by Grim at Penn State’s Association of Religion Data Archives (ARDA) project (three datasets); independent coding by experts at the Hudson Institute’s Center for Religious Liberty using indexes also available from ARDA (one dataset); and content analysis of country constitutions conducted by the Becket Fund for Religious Liberty (one dataset). Pew Forum staff used these for cross-validation. Contrary to what one might expect, therefore, even most countries that limit access to information tend to receive fairly extensive coverage by groups that monitor religious restrictions.
The second key question – the flipside of the first – is whether countries that provide freer access to information receive worse scores simply because more information is available on them. One way to address this issue is to compare the length of U.S. State Department reports on freer-access countries with those of less-free countries. The table below shows the number of words found in the State Department’s 2007 International Religious Freedom report for 18 countries that are chosen for illustrative purposes. As the table shows, the total number of words (73,576) devoted to nine limited-access countries is more than double the number of words (32,508) for nine freer-access countries, with the median number of words approximately three times as large for the limited-access countries as for the open-access countries (7,052 vs. 2,304).
Although this quick comparison shows that problems in freer-access countries are generally not overreported in the U.S. State Department reports, it is the case that some freer-access countries, such as France, the United Kingdom and Germany, do stand out as garnering considerably more coverage than the others on the list; in the case of France, its report is even longer than Saudi Arabia’s. The disproportionate coverage for these three European countries, which general knowledge suggests are less religiously restrictive than the countries on the limited-access list, suggests that a potential over-reporting bias might exist. When one compares the actual results in the table below, however, there appears to be no such problem: The nine limited-access countries have many more reported cases of abuses (17,947) than the freer-access countries (50).
Comparing the GRI scores between the two groups also suggests that the coding methodology overcomes any potential over-reporting bias. Not only do all of the limited-access countries show higher levels of restrictions than any of the freer-access countries, but their average score is more than three times as high (6.8 vs. 2.0).
Only when it comes to religion-related violence and intimidation in society are there more problems reported in the freer-access countries than in the limited-access ones (160 vs. 109). However, the SHI includes several measures – such as Questions No. 8 (Did religious groups themselves attempt to prevent other religious groups from being able to operate?) and No. 11 (Were women harassed for violating religious dress codes?) – that are less susceptible to such reporting bias because they capture general social trends or attitudes as well as specific incidents of violence.
With these limitations in mind, it appears that the coded information on social hostilities is a fair gauge of the situation in the vast majority of countries and a valuable complement to the information on government restrictions. For example, a review of data on Iraq from other studies suggests that even though the level of government restrictions on religion decreased slightly between 2001 and 2007, the level of social hostilities – including sectarian violence, ostracism and physical abuse – steadily increased from 2003 to 2005 and remained at a high level in 2007.
Data on social impediments to religious practice can more confidently be used to make comparisons between countries with sufficient openness, which includes more than nine-in-ten countries covered in the Pew Forum’s coding. An analysis by Grim and Richard Wike, Associate Director of the Pew Research Center’s Global Attitudes Project, tested the reliability of the State Department reports on social impediments to religious practice by comparing public opinion data with data coded from the reports in previous years by Grim and experts at Penn State. They concluded that “the understanding of social religious intolerance embodied in the State Department reports is comparable with the results of population surveys and individual expert opinion.”2
Example of Data Coding: India
Pew Forum coders examined the primary sources to determine whether reported incidents were connected to a specific time, place and perpetrator and to clarify the extent and range of the problems. Looking at data from India helps illustrate this. (Download 4-page PDF.)
Footnotes
1 See “International Religion Indexes: Government Regulation, Government Favoritism, and Social Regulation of Religion” (2006) by Grim and Finke, published in the Interdisciplinary Journal of Research on Religion, Vol. 2 (Article 1). (back to Overview of Procedures)
2 See “Cross-Validating Measures of Global Religious Intolerance: Comparing Coded State Department Reports With Survey Data and Expert Opinion” (forthcoming April 2010) by Brian J. Grim and Richard Wike, to be published in Politics and Religion.
