The excess mortality estimates reported by Kishore et al. for the 102 days after Hurricane Maria depend on the quality of their data. I examined that quality by calculating mortality for the 263 days before Maria and then comparing that rate with those from other sources. Using the authors’ methodology and data, I calculate the mortality as 2.6 deaths (95% CI, 1.4 to 3.8) per 1000 persons from January 1 through September 19, 2017. This rate is inconsistent with the rate calculated on the basis of the official monthly statistics for 2010 through 2017: 8.3 deaths (95% CI, 8.2 to 8.4) per 1000 persons. It is almost statistically impossible for there to have been only 18 deaths in the 3299 households before Maria. The problem remains even when the authors’ later calculations, which adjusted for household size, are taken into account.
Satchit Balsari (2018) Replies to Kane (2018) (our further comments not indented)
It is important to note that our study shed light not only on excess deaths but also on the possibility of prolonged, indirect effects of the hurricane, as well as causes of death and infrastructure disruption. The limitations of the use of post-disaster surveys to calculate mortality or excess deaths are well articulated in the literature. We attempted to address known challenges with sampling, stratification, and biases in the methods we adopted.
These sentences do not answer any aspects of Kane’s critique.
As noted by Kane, the raw, prehurricane estimates of mortality from our survey were lower than historical averages.
This should have been mentioned in the paper. Also, saying pre-hurricane mortality estimates are “lower” is an understatement. In fact, they are 4 standard deviations short of the average and 20.6% below the lowest historical mortality rate.
We have addressed this issue in detail, both in a Technical FAQ (https://github.com/c2-d2/pr_mort_official/blob/master/misc/technical-faq.pdf) and in a public discussion forum with Kane in June 2018 (https://github.com/c2-d2/pr_mort_official/issues/7).
The authors of the paper deserve credit for their transparency and openness with this project.
From Technical FAQ:
As mentioned in the paper, our survey is unable to capture deaths in single-person households. This introduces a bias that results in underestimation of both before and after the hurricane. The death rate in single-person households is found to be 0 — this cannot be true. First, the median age in this single-person households is 69. Second, this death rate is expected because we missed (i.e., could not count) the single-person households where there had been a death (as there was no one to answer the door!)
The paper mentioned that single person households were under-counted but did not explain the reasoning and impact of this issue. This comment raises the fundamental quandary in which the authors’ find themselves. The number of deaths from before the hurricane is implausibly low. Any proposal to “adjust” those rates upwards requires that the same adjustment be applied to the death totals after the hurricane. Any set of adjustments which generate pre-Maria mortality estimates within the historical range would lead to implausibly high post-Maria mortality estimates.
Removing all households with the size of one increases the “before estimate” but not by much. However, it is important to note that the households of size one appeared to be older, on average, than other households which likely results in an underestimate of the mortality rate if we simply remove these from our analysis. An alternative approach would be to plug in a more realistic rate for these households and then adjust for the bias favoring larger households. We took the conservative approach and plugged in the same rate for both before and after the hurricane. To compute a confidence interval we use Keyfitz’s approximation.
| Rate_before | before_lower | before_upper | rate_after | after_lower | after_upper |
|---|---|---|---|---|---|
| 7.72 | 4.16 | 11.3 | 18.2 | 12.4 | 24 |
The above table is from the technical FAQ
Mathematically adjusting the pre-hurricane mortality rate yields a more plausible 7.72 deaths (CI = [4.16, 11.3]). However, this adjustment creates an implausibly high post-hurricane mortality rate of 18.2 per 1000 (CI = [12.6, 24.4]). This implies approximately 30,000 excess deaths. In other words, corpses would be piling up in the streets.
Back to NEJM Reply
We were aware of this issue and did not use prehurricane data for any of the estimates reported in our article.
That is a bug, not a feature, of the paper. It is good that the authors compare various measures from the survey with ACS data. But then why did they not compare the pre-Maria mortality rate from the survey with the official statistics? After all, this is a paper about mortality! There is no more important way to judge the quality of the survey than to see if it got pre-Maria mortality correct. It is statistical malpractice to be aware that your most important variable is completely inaccurate for a period of your study and to keep that fact from your readers.
We also note that when single-person households and age structure are included in the estimate, this low rate is brought within the range of the 2016 death rate for Puerto Rico. The code and data required for this analysis are available at the link to the public discussion forum, above.
Although the adjustment makes the pre-hurricane rate reasonable, the resulting post-hurricane rate is unreasonable.