Search

Begin New Search Publishing Institution Varieties of Democracy Institute (V-Dem) Author Kyle L Marquardt

1. What Makes Experts Reliable?

Author:
Kyle L Marquardt
Publication Date:
06-2018
Content Type:
Working Paper
Institution:
Varieties of Democracy Institute (V-Dem)
Abstract:
Many datasets use experts to code latent quantities of interest. However, scholars have not explored either the factors affecting expert reliability or the degree to which these factors influence estimates of latent concepts. Here we systematically analyze potential correlates of expert reliability using six randomly selected variables from a cross-national panel dataset, V-Dem v8. The V-Dem project includes a diverse group of over 3,000 experts and uses an IRT model to incorporate variation in both expert reliability and scale perception into its data aggregation process. In the process, the IRT model produces an estimate of expert reliability, which affects the relative contribution of an expert to the model. We examine a variety of factors that could correlate with reliability, and find little evidence of theoretically-untenable bias due to expert characteristics. On the other hand, there is evidence that attentive and condent experts who have a basic contextual knowledge of the concept of democracy are more reliable.
Topic:
International Affairs
Political Geography:
Global Focus

2. Experts, Coders, and Crowds: An analysis of substitutability

Author:
Kyle L Marquardt
Publication Date:
11-2017
Content Type:
Working Paper
Institution:
Varieties of Democracy Institute (V-Dem)
Abstract:
Recent work suggests that crowd workers can replace experts and trained coders in common coding tasks. However, while many political science applications require coders to both and relevant information and provide judgment, current studies focus on a limited domain in which experts provide text for crowd workers to code. To address potential over-generalization, we introduce a typology of data producing actors - experts, coders, and crowds - and hypothesize factors which affect crowd-expert substitutability. We use this typology to guide a comparison of data from crowdsourced and expert surveys. Our results provide sharp scope conditions for the substitutability of crowd workers: when coding tasks require contextual and conceptual knowledge, crowds produce substantively different data from coders and experts. We also find that crowd workers can cost more than experts in the context of cross-national panels, and that one purported advantage of crowdsourcing - replicability - is undercut by an insuffcient number of crowd workers.
Topic:
Science and Technology
Political Geography:
Global Focus

3. Constraining Governments: New Indices of Vertical, Horizontal and Diagonal Accountability

Author:
Anna Lührmann, Kyle L Marquardt, and Valeriya Mechkova
Publication Date:
04-2017
Content Type:
Working Paper
Institution:
Varieties of Democracy Institute (V-Dem)
Abstract:
Accountability - constraints on the government’s use of political power - is one of the cornerstones of good governance. However, conceptual stretching and a lack of reliable measures have limited cross-national research and comparisons regarding the role of both accountability writ large and its different sub-types. To address this research gap, we use the V-Dem dataset and Bayesian statistical models to develop new ways to conceptualize and measure accountability and its core dimensions. We provide indices capturing the extent to which governments are accountable to citizens (vertical accountability), other state institutions (horizontal accountability) and the media and civil society (diagonal accountability), as well as an aggregate index that incorporates the three sub-types. These indices cover virtually all countries from 1900 to today. We demonstrate the validity of our new measures by analyzing trends from key countries, as well as by demonstrating that the measures are positively related to development outcomes such as health and education.
Topic:
International Affairs
Political Geography:
Global Focus

4. IRT models for expert-coded panel data

Author:
Kyle L Marquardt and Daniel Pemstein
Publication Date:
01-2017
Content Type:
Working Paper
Institution:
Varieties of Democracy Institute (V-Dem)
Abstract:
Data sets quantifying phenomena of social-scientific interest often use multiple experts to code latent concepts. While it remains standard practice to report the average score across experts, experts likely vary in both their expertise and their interpretation of question scales. As a result, the mean may be an inaccurate statistic. Item-response theory (IRT) models provide an intuitive method for taking these forms of expert disagreement into account when aggregating ordinal ratings produced by experts, but they have rarely been applied to cross- national expert-coded panel data. In this article, we investigate the utility of IRT models for aggregating expert-coded data by comparing the performance of various IRT models to the standard practice of reporting average expert codes, using both real and simulated data. Specifically, we use expert-coded cross-national panel data from the V–Dem data set to both conduct real-data comparisons and inform ecologically-motivated simulation studies. We find that IRT approaches outperform simple averages when experts vary in reliability and exhibit differential item functioning (DIF). IRT models are also generally robust even in the absence of simulated DIF or varying expert reliability. Our findings suggest that producers of cross-national data sets should adopt IRT techniques to aggregate expert-coded data of latent concepts.
Topic:
International Affairs
Political Geography:
Global Focus

Refine your search

more »