DEFINING AND EVALUATING THINK TANKS

Think tanks are public policy research, analysis, and engagement organizations. They are organizations that generate policy-oriented research, analysis, and advice on domestic and international issues that enable policymakers and the public to make informed decisions about public policy issues. Think tanks may be affiliated with political parties, governments, interest groups, or private corporations or constituted as independent non-governmental organizations (NGOs). These institutions often act as a bridge between the academic and policymaking communities, serving the public interest as an independent voice that translates applied and basic research into a language and form that is understandable, reliable, and accessible for policymakers and the public.

Structured as permanent bodies, in contrast with ad hoc commissions or research panels, think tanks devote a substantial portion of their financial and human resources to commissioning and publishing research and policy analysis in the social sciences: political science, economics, public administration, and international affairs. The major outputs of these organizations are books, monographs, reports, policy briefs, blogs, conferences, seminars, web-based reports and commentary, formal briefings and informal discussions with policymakers, government officials, and key stakeholders.

While think tanks may perform many roles in their host societies, not all think tanks do the same things to the same extent. Over the last 85 years, several distinctive organizational forms of think tanks have come into being that differ substantially in terms of their operating styles, their patterns of recruitment, their aspirations to academic standards of objectivity and completeness in research and their engagement of policy makers, the press and the public.

Think Tank Typology/Think Tank Affiliations

1. Autonomous and Independent : Significant independence from any one interest group or donor and autonomous in its operation and funding from government.
2. Quasi Independent:  Autonomous from government but an interest group (i.e. unions, religious groups, etc.), donor or contracting agency provides a majority of the funding and has significant influence over operations of the think tank.
3. University Affiliated : A policy research center at a university.
4. Political Party Affiliated : Formally affiliated with a political party.
5. Government Affiliated : A part of the structure of government.
6. Quasi Governmental : Funded exclusively by government grants and contracts but not a part of the formal structure of government.
7. For Profit : Public policy research unit located within a corporation or operating as a free standing for-profit think tank.

Evaluation Criteria

• The quality and commitment of the think tank’s leadership (Chief Executive and governing body). This involves effectively managing the mission and programmes of the think tank, mobilizing the financial and human resources necessary to fulfill the mission and monitoring the quality, independence and impact of the think tank;
• The quality and reputation of the think tank’s staff. Ability to assemble a critical mass of highly skilled, experienced and productive scholars and analysts who are recognized as either emerging or established experts in their respective area of research;
• The quality and reputation of the research and analysis produced. The ability to produce high quality, rigorous, policy oriented research that is accessible to policymakers, media and the public;
• Ability to recruit and retain elite scholars and analysts;
• Academic performance and reputation. This involves the academic rigor associated with the research conducted. This includes formal accreditation of a think tank’s scholars and analysts, the number and type of scholarly publications produced such as: books, journals and conference papers and the number of presentations delivered at scholarly and other professional meeting and the number and type of citations of the think tanks scholars’ research in scholarly publications produced by other scholars;
• The quality, number, and reach of its publications;
• The impact of a think tanks research and programmes on policymakers and other policy actors. Policy recommendations considered or actually adopted by policymakers, civil society or policy actors;
• Reputation with policymakers (name recognition associated with specific issues or programmes, number of briefings and official appointments, number of policy briefs and white papers produced, legislative testimony delivered);
• A demonstrated commitment to producing independent research and analysis. This involves standards and policies for producing rigorous evidence based research and analysis that are posted and monitored by the organization, research teams and individual researchers. This includes disclosure of conflict of interest (financial, institutional or personal) and a commitment to non-partisanship and established professional standards for research in the social sciences;
• Access to key institutions. The ability to reach and connect with key audiences and personnel such as government officials (elected and appointed), civil society, traditional and new media, and academia;
• Ability to convene key policy actors and to develop effective networks and partnerships with other think tanks and policy actors;
• Overall output of the organization (policy proposals, web visits, briefings, publications, interviews, conferences, staff nominated to official posts);
• Utilization of research, policy proposal and other products. The effective transmission and utilization of policy briefs, reports, policy recommendations and other products by policymakers and the policy community, number of current and former staff serving in advisory roles to policymakers, advisory commissions, etc., awards given to scholars for scholarly achievement or public service;
• Usefulness of organization’s information in public engagement, advocacy work, preparing legislation or testimony, preparing academic papers or presentations, conducting research or teaching;
• Ability to use electronic, print and the new media to communicate research and reach key audiences;
• Media reputation (number of media appearances, interviews and citations);
• Ability to use the Internet including social media tools, to engage with policymakers, journalists and the public;
• Web Site and Digital presence. The quality, accessibility, effective maintenance of the organization’s web presence, as well as, the quality and level Digital traffic and engagement. (quality, accessibility and navigability of web site, number of website visitors, page views, time spent on pages, “likes” or followers);
• Level, diversity and stability of funding. The ability of an organization to mobilize the necessary financial resources to support and sustain the think tank over time (endowment, membership fees, annual donations, government and private contracts, earned income);
• Effective management and allocation of financial and human resources. The ability of a think tank to effectively manage its money and people so that they produce high quality outputs that achieve maximum impact;
• Ability of the organization to effectively fulfill the terms of the gifts, grants and contracts from government(s), individuals, orporations and foundations who have provided the financial support to the think tank (financial stewardship);
• The organization’s ability to produce new knowledge, innovative policy proposals or alternative ideas on policy;
• Ability to bridge the gap between the academic and policymaking communities;
• Ability to bridge the gap between policymakers and the public;
• Ability to include new voices in the policymaking process;
• Ability of organization to be inscribed within issue and policy networks;
• Success in challenging the traditional wisdom of policymakers and in generating innovative policy ideas and programmes;
• The impact on society. Direct relationship between the organization’s efforts in a particular area to a positive change in societal values such as significant changes in the quality of life within respective country (amounts of goods and services available to citizens, state of physical and mental health, quality of environment, quality of political rights, access to institutions).

Clearly, assessing the impact of think tanks is not an easy endeavor to undertake given the various and conflicting actors, events, and politics involved in the policy making process. Despite the significant challenges in establishing a causal relationship between knowledge and policy, it is necessary for think tanks to understand and effectively respond to the growing chorus of questions being raised by donors, journalists, and the public about the role and influence of think tanks in civil societies and governments around the world.

Think tanks can employ a variety of metrics to assess the impact including such measures as an increase in research and analysis they produce as well as to account for their contributions to the policymaking environment and civil society.

The metric provided below is designed to serve as a catalyst for a discussion on how to effectively measure the impact of think tanks.
• Resource Indicators : Ability to recruit and retain leading scholars and analysts; the level, quality, and stability of financial support; proximity and access to decision-makers and other policy elites; a staff with the ability to conduct rigorous research and produce timely and incisive analysis; institutional currency; quality and reliability of networks; and key contacts in the policy academic communities, and the media
• Utilization Indicators : Reputation as a “go-to” organization by media and policy elites in the country; quantity and quality of media appearances and citations, web hits, testimony before legislative and executive bodies; briefings, official appointments, consultation by officials or departments/agencies; books sold; reports distributed; references made to research and analysis in scholarly and popular publications and attendees at conferences and seminars organized
• Output Indicators : Number and quality of: policy proposals and ideas generated; publications produced (books, journal articles, policy briefs, etc.); news interviews conducted; briefings, conferences, and seminars organized; and staff who are nominated to advisory and government posts
• Impact Indicators : Recommendations considered or adopted by policymakers and civil society organizations; issue network centrality; advisory role to political parties, candidates, transition teams; awards granted; publication in or citation of publications in academic journals, public testimony and the media that influences the policy debate and decision-making; listserv and web site dominance; and success in challenging the conventional wisdom and standard operating procedures of bureaucrats and elected officials in the country

Beyond this quantitative assessment, an effective evaluation of impact should also involve NGOs, as well as members of the government and policymakers, to ascertain the degree to which they have utilized the grantee’s research output. This participation can be obtained through interviews, surveys, questionnaires, and focus group meetings, utilizing the Outcome Mapping which moves away from assessing the products of an activity or a programme to focus on changes in behaviours and relationships (outcomes) which can lead to changes.” Impact can be viewed as positive if it “changes the behavior, relationships, activities, or actions of the people, groups, and organizations with whom a programme works directly.

Although this qualitative assessment is essential because it recognizes that policy impact can be successfully achieved even if policy prescriptions are not directly translated into actual policy, this assessment should be translated into numerical rankings, thereby allowing comparisons with baseline data for effective monitoring and evaluation in the future.