Survey Rake Weights

Surveys

POL 307

Rake weights are one way of adjusting for survey nonresponse by iteratively updating the weights one variable at a time until they converge. The basic methodology is as follows:

Before starting, identify the population proportion in each category of each weighting variable, and set every respondent’s weight to 1.

For the first variable, calculate each category’s sample proportion using the current weights.
For each category, calculate its adjustment factor: population proportion divided by weighted sample proportion.
Multiply each respondent’s current weight by the adjustment factor for their category.
Move to the next variable and repeat Steps 1–3 using the updated weights.

Adjusting all variables completes one full pass. Then return to Step 1 and begin another pass using the weights you’ve just calculated.

We repeat this process because adjusting for one variable can move the distribution of another variable further from its population value. We correct this by readjusting for that variable on the next pass. Each pass will generally require smaller and smaller adjustments.

We stop when the weighted sample proportions change by less than a chosen tolerance between passes—for example, 0.1 percentage points. At this point, the weights are said to have converged.

Example

Table 1 shows the education, age, and partisan breakdown in a simulated population and in our final sample. As is common, college-educated and older voters were much more likely to respond. Because partisanship is related to both age and education, our estimate of partisanship is also biased. We can reduce this bias by using rake weights to adjust on the Age and Education variables.

Table 1

(a) Population

Category	Percent
No College	66%
College	34%
Young	30%
Middle	50%
Old	20%
Democrat	36%
Independent	25%
Republican	39%

(b) Sample

Category	Percent
No College	32%
College	68%
Young	19%
Middle	46%
Old	34%
Democrat	44%
Independent	18%
Republican	37%
N	984

Initial Setup

As described above, raking is an iterative process. Table 2 sets up the beginning of the process. It includes the population proportions for the variables used in the weights. It also shows the sample proportion of each age category, calculated using an initial weight of 1 for every observation. Finally, the adjustment is simply \(\frac{\text{Pop}}{\text{Sample}}\). The rows below show four observations from the data so we can track how their weights change during the process.

Table 2: Initial Weights Setup

	Initial	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.1941	0.4624	0.3435	-	-
Adjustment	-	1.5476	1.0800	0.5828	-	-
No College-Middle	1.0000	-	-	-	-	-
No College-Young	1.0000	-	-	-	-	-
No College-Young	1.0000	-	-	-	-	-
College-Middle	1.0000	-	-	-	-	-

First Variable: Age

We will start by adjusting on the Age variable. We could adjust all Age categories at once, but to make the process clearer, we will go through them one at a time. The first category is Young: the population target is 30.04%, while the current sample estimate is 19.41%. The adjustment factor is therefore 1.548. To make this adjustment, we multiply the current weight (1 for everyone) by this factor for every Young respondent and leave the other weights unchanged. Table 3 shows the new weights in the Age: Young column.

Table 3: Adjusted for Age - Young Once

	Initial	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.1941	0.4624	0.3435	-	-
Adjustment	-	1.5476	1.0800	0.5828	-	-
No College-Middle	1.0000	1.0000	-	-	-	-
No College-Young	1.0000	1.5476	-	-	-	-
No College-Young	1.0000	1.5476	-	-	-	-
College-Middle	1.0000	1.0000	-	-	-	-

Next, we adjust the Middle category of Age. Here the adjustment factor is closer to 1: 1.080. Again, we apply it only to respondents in the Middle category, as shown in Table 4.

Table 4: Adjusted for Age - Middle Once

	Initial	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.1941	0.4624	0.3435	-	-
Adjustment	-	1.5476	1.0800	0.5828	-	-
No College-Middle	1.0000	1.0000	1.0800	-	-	-
No College-Young	1.0000	1.5476	1.5476	-	-	-
No College-Young	1.0000	1.5476	1.5476	-	-	-
College-Middle	1.0000	1.0000	1.0800	-	-	-

Next, we repeat this for the Old category. In this case, we adjust the weights down by multiplying them by 0.583. None of the example rows are in this category, so their weights do not change, although the weights of Old respondents elsewhere in the data are adjusted (Table 5).

Table 5: Adjusted for Age - Old Once

	Initial	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.1941	0.4624	0.3435	-	-
Adjustment	-	1.5476	1.0800	0.5828	-	-
No College-Middle	1.0000	1.0000	1.0800	1.0800	-	-
No College-Young	1.0000	1.5476	1.5476	1.5476	-	-
No College-Young	1.0000	1.5476	1.5476	1.5476	-	-
College-Middle	1.0000	1.0000	1.0800	1.0800	-	-

Second Variable: Education

Now that we have adjusted for Age, we recalculate the proportion of the sample in each Education category, this time using the weights we just created. After the Age adjustments, 30.92% of the weighted sample has no college degree and 69.08% has a college degree. These numbers are only slightly different from the initial estimates of 31.61% and 68.39%. Table 6 shows these proportions and the new adjustment factors.

Table 6: Setup for Education Adjustment

	Initial	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.1941	0.4624	0.3435	0.3092	0.6908
Adjustment	-	1.5476	1.0800	0.5828	2.1452	0.4874
No College-Middle	1.0000	1.0000	1.0800	1.0800	-	-
No College-Young	1.0000	1.5476	1.5476	1.5476	-	-
No College-Young	1.0000	1.5476	1.5476	1.5476	-	-
College-Middle	1.0000	1.0000	1.0800	1.0800	-	-

We apply these adjustments the same way as before. This time, however, we multiply the adjustment factors by the weights we just created rather than by the initial weights of 1. In Table 7, we multiply the weights for the first three example rows by 2.145.

Table 7: Adjusted for Education - No College Once

	Initial	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.1941	0.4624	0.3435	0.3092	0.6908
Adjustment	-	1.5476	1.0800	0.5828	2.1452	0.4874
No College-Middle	1.0000	1.0000	1.0800	1.0800	2.3169	-
No College-Young	1.0000	1.5476	1.5476	1.5476	3.3199	-
No College-Young	1.0000	1.5476	1.5476	1.5476	3.3199	-
College-Middle	1.0000	1.0000	1.0800	1.0800	1.0800	-

Next, we adjust the weights for college-educated respondents (the second Education category). The last column of Table 8 shows that each weight now depends on the respondent’s category for both Age and Education. Two observations have the same values on both variables, so their weights are the same. The other two observations are both middle-aged but differ in education, so their weights also differ.

Table 8: Adjusted for Education - College Once

	Initial	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.1941	0.4624	0.3435	0.3092	0.6908
Adjustment	-	1.5476	1.0800	0.5828	2.1452	0.4874
No College-Middle	1.0000	1.0000	1.0800	1.0800	2.3169	2.3169
No College-Young	1.0000	1.5476	1.5476	1.5476	3.3199	3.3199
No College-Young	1.0000	1.5476	1.5476	1.5476	3.3199	3.3199
College-Middle	1.0000	1.0000	1.0800	1.0800	1.0800	0.5264

Second Pass

We’ve now completed one full pass, but we are not done. Adjusting for education after age has moved the age distribution away from its targets. For example, the weights after the first pass give an estimate of 26.11% young people, which is close to the population target of 30.04% but not exactly right. Because of this, we need to readjust on Age. The adjustments and calculations are in Table 9.

Note

In this simulated population, education and age are independent, so the adjustments do not affect each other very much. The more strongly the weighting variables are associated with each other, the more adjusting for one will affect the distribution of another.

Table 9: Setup for Second Age Adjustment

	First Pass	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.2611	0.5382	0.2007	-	-
Adjustment	-	1.1503	0.9279	0.9976	-	-
No College-Middle	2.3169	-	-	-	-	-
No College-Young	3.3199	-	-	-	-	-
No College-Young	3.3199	-	-	-	-	-
College-Middle	0.5264	-	-	-	-	-

In Table 10, we make these adjustments, slightly increasing the weights of young and old respondents while decreasing the weights of middle-aged respondents.

Table 10: Second Age Adjustment

	First Pass	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.2611	0.5382	0.2007	-	-
Adjustment	-	1.1503	0.9279	0.9976	-	-
No College-Middle	2.3169	2.3169	2.1499	2.1499	-	-
No College-Young	3.3199	3.8191	3.8191	3.8191	-	-
No College-Young	3.3199	3.8191	3.8191	3.8191	-	-
College-Middle	0.5264	0.5264	0.4885	0.4885	-	-

We use these weights to re-estimate the proportions of respondents with and without a college degree and then adjust for Education. Table 11 shows this complete step. Again, the adjustments are very small.

Table 11: Second Education Adjustment

	First Pass	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.2611	0.5382	0.2007	0.6578	0.3422
Adjustment	-	1.1503	0.9279	0.9976	1.0083	0.9840
No College-Middle	2.3169	2.3169	2.1499	2.1499	2.1678	2.1678
No College-Young	3.3199	3.8191	3.8191	3.8191	3.8509	3.8509
No College-Young	3.3199	3.8191	3.8191	3.8191	3.8509	3.8509
College-Middle	0.5264	0.5264	0.4885	0.4885	0.4885	0.4806

Third Pass and Convergence

We’ve now completed two full passes of raking, so we start again at the beginning. Table 12 shows that the adjustment factors are essentially 1 for every age category, so the weights appear to have converged quickly. There are several ways to define convergence, and the details are beyond the scope of this guide. In short, we choose a rule for when the weights are “good enough” and additional passes no longer make meaningful improvements.

Table 12: Setup for Third Age Adjustment

	Second Pass	Age: Young	Age: Middle	Age: Old	Ed: No	Ed: Col
Pop	-	0.3004	0.4994	0.2002	0.6633	0.3367
Sample Estimate	-	0.2997	0.5000	0.2002	-	-
Adjustment	-	1.0022	0.9988	0.9998	-	-
No College-Middle	2.1678	-	-	-	-	-
No College-Young	3.8509	-	-	-	-	-
No College-Young	3.8509	-	-	-	-	-
College-Middle	0.4806	-	-	-	-	-

Converged Rake Weights

Table 13 compares the population (Table 13 (a)), the initial sample (Table 13 (b)), and the sample weighted after three passes (Table 13 (c)). On the weighting variables, the weighted sample is now within 0.01 percentage points of the population. The partisanship estimates are also much closer to the population values, differing by less than 2 percentage points. This is a best-case scenario: in this simulated data, we know what caused nonresponse (education and age), we know that these variables are related to partisanship, and we have population targets for age and education.

Table 13: Margins Compared

(a) Population

Cat	Percent
No College	66.33%
College	33.67%
Young	30.04%
Middle	49.94%
Old	20.02%
Democrat	36.33%
Independent	24.85%
Republican	38.82%

(b) Initial Sample

Cat	Percent
No College	31.61%
College	68.39%
Young	19.41%
Middle	46.24%
Old	34.35%
Democrat	44.41%
Independent	18.39%
Republican	37.20%

Cat	Percent
No College	66.32%
College	33.68%
Young	30.04%
Middle	49.94%
Old	20.02%
Democrat	34.34%
Independent	26.35%
Republican	39.30%

One last point about the limits of rake weights: the weighted estimates match the population margins (the overall distributions of education and age), but there is no guarantee that they will match the joint distribution of those variables. Table 14 shows the percentage of the population and weighted sample in each age-by-education cell. The numbers are fairly close, but they do not match exactly. This is partly because the simulation made one cell (young \(\times\) no college) especially unlikely to respond. Raking on the margins does not directly account for this kind of interaction in nonresponse, although you can account for some interactions by raking on combined categories (for example, age-by-education cells) when population targets for them are available.

Table 14: Crosstabs

(a) Population

Population
Age	No College	College
Young	20.03%	10.01%
Middle	33.24%	16.70%
Old	13.06%	6.96%

(b) After Three Rakes

Three Rakes
Age	No College	College
Young	17.26%	12.78%
Middle	35.65%	14.29%
Old	13.42%	6.60%