Records Request #03

Index to New York State Deaths (Outside of New York City), 1880-1956

The digital versions of the approximately 1580 microfiche sheets, which yielded tens of thousands of scanned images and several million names, are now online and free for public use, forever! https://archive.org/details/nydeathindex

On January 4, 2016, Reclaim The Records submitted a new York State Freedom of Information Law (FOIL) request to the New York State Department of Health for a copy of the entire New York State death index, from June 1880 through December 31, 1956. We requested the index only in this FOIL request.

And seventeen months later, we won the records!

New York State death index - image #1 of 2

This death index covers 1880-1956, which is seventy-seven years! Here are some things to keep in mind:

  1. 1880 and 1881 have very limited data, as the state law requiring reporting deaths to the state didn’t have great compliance at first.
  2. 1943 is, unfortunately, extremely hard to read, and we don’t know if any better copy exists. If there is, we’ll go after it with a new FOIL request and scan it.
  3. New York City deaths are, for the most part, not included in this statewide death index. That’s because New York City and New York State are completely different vital records jurisdictions. However, some deaths that were in neighborhoods that are part of modern-day New York City — such as Flushing (Queens) or Canarsie (Brooklyn) — do show up in the pre-1897/1898 statewide death index! That’s because they weren’t incorporated as part of the city yet at the time they were recorded.
  4. Three major cities are not included in this statewide death index until 1914 or 1915: Yonkers, Buffalo, and Albany. Reclaim The Records has made three new and separate FOIL requests to those city clerks for the first-ever public copies of their 1880-1914 death indices, in order to complete the missing data.

We uploaded all these files, year by year, to the Internet Archive (archive.org). Other websites, as well as individual researchers, are welcome to use, download, and even host the files on their own websites, and create transcription projects if they wish. The files are entirely in the public domain, free to use without any restrictions. If you do create a transcription, though, we’d appreciate just having our name mentioned, and a link back to our website (https://www.ReclaimTheRecords.org/).

Each year’s worth of files can be very large, perhaps a 10 GB zip file for images for an earlier year, or 20 GB for later years, as the later years do have more names and therefore more images in them. We would recommend browsing through the data using the images hosted online on the Internet Archive’s website, rather than trying to download the huge zip files to your computer on a slow home modem connection, but that’s up to you.

Here’s what the sample URL structure looks like:

https://archive.org/details/New_York_State_Death_Index_1880
https://archive.org/details/New_York_State_Death_Index_1881
https://archive.org/details/New_York_State_Death_Index_1882
https://archive.org/details/New_York_State_Death_Index_1883
https://archive.org/details/New_York_State_Death_Index_1884
https://archive.org/details/New_York_State_Death_Index_1885

…and so on. You get the idea, just change the year at the end of the URL to any year between 1880 and 1956.

The best way to see the WHOLE LIST of the items in the New York State Death Index collection is through this direct link:
https://archive.org/details/nydeathindex

The full text of our request is available below, and also available to follow on the MuckRock website:

To Whom It May Concern:

Pursuant to the New York State Freedom of Information Law (1977 N.Y. Laws ch. 933), I hereby request the following records:

I would like to receive a copy of the New York State death index, from June 1880 (or as early as such records are available) through December 31, 1956, inclusive. Please note that this request is for the basic index only, and not the actual death certificates.

The New York State Department of Health has already made this index freely available in microfiche format at various public libraries in upstate New York, as well as at the New York City branch of NARA, the National Archives and Records Administration. However, that microfiche is only accessible to researchers who are physically onsite at one of those locations.

The New York State Department of Health has also made a selection of data from the index for the years 1957-1965 available to the public in database format on the New York State open data portal, at this online address:
https://health.data.ny.gov/Health/Genealogical-Research-Death-Index-Beginning-1957/vafa-pf2s

That data is available for searches online and for download to a personal computer. The data is updated quarterly for the more recent years with a fifty year lag; e.g. the information from 1966 will start to be added to that file in 2016.

If the more modern data from 1957-1965 is already available to the public, then there doesn’t appear to be any reason that the older data from 1880-1956 could not be made available too, in whatever format it currently exists.

This request is not being made for commercial purposes. The requested documents will be scanned and uploaded to the Internet, and will be made freely available to the general public. It is anticipated that some non-profit genealogical groups may choose to transcribe the information in the death index, to turn it into a new text-searchable database. We would be happy to share any such database with the Department of Health.

If possible, I would prefer the request filled electronically, by e-mail attachment if available or via other media. Please be advised that if any microfiche or microfilm copies need to be made, these will also require shipping fees to California. Please inform me of any potential charges in advance of fulfilling my request.

Please also be advised that this FOIL request is being filed publicly through the website MuckRock.com, and all correspondence about this request will be immediately published to the general public.

Thank you in advance for your anticipated cooperation in this matter. I look forward to receiving your response to this request within 5 business days, as the statute requires.

And here’s the story about how we did it. It’s kind of long, but it’s really juicy, so if reading tales of government malfeasance is your thing, settle in for a treat.

 

Part I: We ask nicely

Back in January 2016, Reclaim The Records wrote and sent a simple Freedom of Information Law request to the New York State Department of Health (DOH). You can read the whole thing online if you want, because we publish all our requests and correspondence out in the open, usually through the website MuckRock, which helps people send and organize FOIA and state FOI requests.

We made two main points to the DOH in our request:

  1. This 1880-1956 death index we’re asking for is already a public record, because you guys literally published for the public it years ago on microfiche and gave the copies to a few public libraries, plus the NARA branch in Manhattan. But microfiche is an old-fashioned format and hard to use, and the library copies are kind of scratched up. And a lot of genealogists with New York roots live somewhere else, and don’t want to have to travel across the country to do look-ups. Why can’t we just get a copy of that microfiche set scanned, so we can all do research at home, in our pajamas?
  2. You guys at the DOH already transcribed and publish the 1957-1965 (now 1967) death index data on your own DOH open data portal, so what about the pre-1957 stuff? Why can’t genealogists get that? If all this mid-century data is okay with you, clearly not “too personal” or intrusive, then the very old data must be totally fine, right?

And so we sent off the request. And we waited to hear back.

Finally, on March 29, 2016, they gave us a response to our request. And what they had to say was shocking. Yes, the Department of Health conceded that we had the legal right to a copy of the statewide death index under the New York Freedom of Information Law. They could find no reason to legally deny our request.

So instead, they tried another way to dissuade us, by refusing to take the subsequent step in the state law, which was for the agency to estimate the actual costs of making those copies.

 

Part II: The state breaks the law…like, a lot

Rather than count up the actual number of microfiche sheets in question, or to provide a fair estimate for the costs of digitally scanning each of those sheets, the DOH simply decided that there would be a flat fee of $2,000 per year of microfiche copied, regardless of how many sheets were actually included in that year.

Since the microfiche set covered 1880-1956, which is seventy-six years, this meant that the DOH wanted us to pay them $152,000 for the New York State death index.

And they weren’t done yet! They also stated in this response that we had a maximum of ten business days to give them a yes or no answer to this “estimate”.

And we must then immediately provide them with a 50% downpayment, meaning $76,000. Otherwise, they said that they would close and cancel this FOIL request entirely.

(You can probably imagine what went through our head when we read their letter with these demands.)

(And you can read it for yourself, because we posted their letter to MuckRock.)

So we did what any good activist group would do, and called our attorneys at Rankin & Taylor, the same awesome bunch who had fought and won our right to the New York City marriage license index the year before.

 

Part III: Bring on the lawyers

Our attorneys confirmed for us that the state’s response was illegal in multiple ways. First of all, an agency that grants a FOIL request must tell you how approximately many records you’ve actually won. Here, New York was claiming that there was an unknown number of microfiche sheets per year but that the number was somehow exactly the same for each of the seventy-six years. This was absurd, as of course the number of deaths in the state varied from year to year, and grew over time as the population increased, and sometimes had unusual spikes in the death rate as happened in the 1918-1919 influenza pandemic. For the state to simply make a flat fee for each of the seventy-six years in question, even if that fee had been reasonable, was breaking the law all on its own.

But of course, that fee wasn’t reasonable, either. The cost of standard paper copies is capped in New York’s FOIL at twenty-five cents a page, but we were asking for microfiche sheets whose copying costs were not explicitly mentioned in the statue; the law just said that fees for copies of other types of non-paper records may be charged based upon the actual cost of reproduction. But the state did not provide any information on why it thought a year’s worth of microfiche scanning would cost $2,000. They did not provide any copies of estimates from vendors, nor the names of vendors, nor published price lists, nor did they ever provide some after we pressed them for details. They invented the $2,000 per year cost, and then never explained or apologized for it.

Thanks to a genealogist friend who happened to be in Albany and who took it upon herself to count up all the microfiche sheets of the death index at the library, we knew there were 1,522 microfiche sheets in total in the state death index, give or take the few sheets that had gone astray over the years.

Based on quotations from multiple digitization companies whose price lists we found online, we were able to establish that the actual costs of scanning each microfiche sheet would be about $2 per sheet, maybe a little less if we were doing a big bulk order. That meant that the actual cost of fulfilling this FOIL request should have been about $3,044 dollars for the full seventy-six years of microfiche scanning, plus a little more for the cost of shipping.

Even the National Archives (NARA), whose prices tend to be some of the highest amongst archives who directly create scans and copies for researchers, say they will prepare microfiche scans for researchers for $12 per sheet, although they also include free shipping. Even at that inflated cost, our death index project would have cost about $18,264 in total – expensive, yes, but still far, far less than Albany’s “estimate” of $152,000! (For the record, the current median price of a home in Albany is $166,200.)

Finally, our attorneys confirmed for us that the Department of Health’s demand that we provide them a 50% downpayment for their made-up $152,000 estimate within ten business days – a demand known as an “exploding offer” – was entirely illegal. There is no such requirement in New York State’s FOIL that a records request be agreed to in a specific timeframe, much less in one so short a window. The Department’s Records Access Officer had completely made it up. And they never apologized for it, even after they dropped that part of the demand when our attorney called them out on their behavior.

Our attorney also set up a phone conversation with the pro bono New York State Committee on Open Government, a watchdog group funded by the New York legislature, and their executive director confirmed for us that the Department of Health was breaking FOIL and acting in bad faith. Furthermore, the Department’s Records Access Officer had exceeded a reasonable or legal timeframe for even making this wholly ridiculous response. They had responded 143 days after our original request, and fifty-three days after we had repeatedly asked them to justify their outrageous fees; New York’s FOIL requires a full government response in fivebusiness days and provides for a twenty business day limit in unusual circumstances.

This kind of legal trickery, first a delay and then a refusal to provide a real estimate of the records’ reproduction costs, was insidious, because on the surface, the state could act innocent and claim that they hadn’t really blocked off our legal access to the records, it was just that we had been unable or unwilling to pay for them. It was essentially a non-denial denial, or a “constructive denial” as the legal terminology calls it – but getting a Department of Health watchdog to side with us instead of their own agency would be difficult. And that’s just what happened.

For reasons that defy explanation, when we filed a formal appeal of this FOIL request with the Department of Health on June 3, 2016, the person to whom we appealed, who also worked for the Department, proceeded to deny our appeal.

 

Part IV: FOIL request, take two

At that point, we had a choice to make. We could take the next step and take the case to trial, suing the state of New York. We would probably win, but it would be expensive, and there was no guarantee that we would win back attorneys fees even if we did win all the records. In New York, the awarding of attorneys fees is left up to the discretion of the judge, and is a separate question from whether the records were wrongly being withheld.

But we had another option. In all the months of fighting with the DOH for these records, our point person on the case had been their Records Access Officer, a woman whose name we will omit here but which can be easily determined from reading through our legal paperwork, which are all a matter of public record and all online. She had been the person behind the lack of transparency on pricing, the ten-day exploding offer, the months of delays, and all of that.

And as we were approaching the legal deadline of whether or not to pursue legal action against the state, we heard through the grapevine that this woman was suddenly no longer the Records Access Officer. We hadn’t yet had time to file a formal legal complaint against her and her office’s actions; perhaps she had gotten into trouble for unethical and illegal behavior with other records requestors? In any case, she was now gone.

This meant that instead of suing the state, we could try a different tactic. We could re-file a new FOIL request for the death index, starting from scratch, with the new Records Access Officer, hoping that this one would be more reasonable than her predecessor. And that’s just what we did. Luckily, this new one was far more pleasant to deal with, and as far as we could see she did not do anything overtly illegal.

 

Part V: Can we scan that for you?

At this point, it was nearing the end of 2016. In an effort to move things along, we asked the new Records Access Officer if we could bring in our own digitization vendor to do the scanning of the microfiche sheets they held, just to make it easier on the state and get these records copied already. After all, they’d already said on their own letterhead that we were legally entitled to a copy of the files; we were now basically just nitpicking over how much it would cost and who would pay for it.

She replied that we could indeed hire our own vendor to do the scanning…with a few minor restrictions. First of all, our vendor would have to come onsite to Albany, and could not remove the sheets from the state facility. Second, the vendor would only be allowed to do their work there during business hours on weekdays and when employees of the Department had free time to supervise them doing the work directly. Third, they could only do the microfiche scanning sheet by sheet, hand-fed into the machine instead of using the automatic feeder, a process that would take much longer, and therefore incur a higher cost. Fourth, we had to provide a full spec list and product manual of the brand and exact model of the scanner that our vendor would use, in order to get the approval from the state. And so on.

Frustrated but persistent, we said yes to all these demands. Yes, we’ll agree to your picky timetable; yes, we’ll get you all the details on our magic microfiche scanner. But can we just have the death index already? Please?

 

Part VI: New York hires an archivist

Well, no. Because the next hoop the state wanted us to jump through was this: we were told that we now had to wait a few more months while the state hired an archivist to come to Albany and assess the quality of the microfiche copies of the death index they held in their state vaults, to see if they were too fragile to scan with this magic microfiche machine. And that meant that we also had to wait while they discussed and approved the government funding to hire this archivist.

So while we waited, we tried yet another tack to speed up this request, which was by now more than a year old. Hey, remember how there are about eleven libraries in New York State where the public can use a copy of this microfiche? Sure, those copies of the microfiche are in much worse physical shape than the ones in the Department’s vaults, but they’re available right now. Can we maybe scan one of those copies instead of waiting for you guys in Albany to get an archivist to assess your copies? Wouldn’t that be faster and easier?

 

Part VII: Nope, not ours

And here’s one of the nuttiest things that happened in this whole story: the New York State Department of Health then repeatedly denied to us that these copies of their own microfiche, which are in libraries across the state through a strict legal arrangement between the Department and those libraries, ever belonged to them in the first place. They claimed that the libraries are under the Department of Education, and to get access to their holdings, including these death index microfiche, we would have to file a FOIL request with the Department of Education instead of the Department of Health.

In other words, we were the ones who had to inform the Department of Health about their own property, their own microfiche copies of the statewide death index, being available throughout the state. And even when we did, they repeatedly denied to us in print and by phone that these microfiche in their libraries are their property. So we decided not to pursue the idea of duplicating the libraries’ copies of the fiche, and to wait out the arrival of the new archivist to confirm that the state’s own copies were in acceptable condition.

 

Part VIII: Free at last?

And then? Another bombshell dropped – but this time, a good one, or so it seemed, The Records Access Officer, on one of our many phone calls to her asking for a status update, suddenly let us know that the Department would be hiring a vendor to scan the vault copies. And the state would not charge us anything for the vendor’s labor. They didn’t even give us an estimated cost at all.

This was wonderful news, but also quite odd – hadn’t we just been spending months with the previous Records Access Officer fighting over the cost of the scanning? But now it was all going to be handed to us for free? We didn’t want to look a gift horse in the mouth, so we didn’t ask too many questions. And a few months later, seventeen months after the original FOIL request had been filed, we finally received in the mail a small USB hard drive containing all the seventy-six years of scanned microfiche. It was over, Reclaim The Records had the New York State death index.

 

Part IX: There’s no such thing as a free lunch

Several months later, we found out why Reclaim The Records had never received a bill, or even an estimate, from the state itself or from the mysterious vendor who had scanned the microfiche sheets on behalf on the state.

The vendor turned out to be none other than Ancestry.com, the world’s largest for-profit genealogy company. They, along with every other genealogy organization in the world, as well as those from many other industries like insurance companies, had been hammering on New York State for years, for decades, begging for copies of the incredibly valuable data of this statewide death index. But not one of these companies or organizations had ever cracked the wall…until Reclaim The Records showed up on the scene with our awesome new approach of using Freedom of Information laws to force the state’s hand.

And at that point, Ancestry apparently felt threatened and did not want to be cut out of the records pipeline entirely. Even though Reclaim The Records always has and always will post all our data online for free, whether images or text, clearly in the public domain, it seems like Ancestry wanted to make sure they still were a player in the game with unique access to the state. And so unbeknownst to us, they had gone to the state and said look, Reclaim The Records is gunning for you guys, they are clearly going to win their legal case against you, how about you let us at least scan the data for you for free?

On the one hand, this was a very generous thing for Ancestry to do, as it meant that our little non-profit organization didn’t have to bear the cost of thousands of dollars for scanning, which the state was certainly allowed to charge us under FOIL. But there was also an ulterior motive: by volunteering to be the vendor doing the actual scanning, Ancestry was guaranteeing that they had access to the raw images of the statewide death index before anyone else, even before the state itself. This meant that they had a head start on scanning and perhaps even indexing this brand new data and preparing it for upload to their website, while we at Reclaim The Records, the original FOIL requestor, waited ignorantly and patiently for our own copy to arrive in the mail.

It also meant that a few months later, when Ancestry had finally transcribed the entire New York State death index and revealed it on their website as a fait accompli, they put their copy of both the images and the new searchable database behind their paywall. That would have been fine with us, except for one little issue: they never once mentioned how they acquired that data in the first place. They presented it entirely as their own achievement.

 

Part X: Ancestry takes the credit

Ancestry never mentioned that Reclaim The Records had initiated the FOIL request, had fought the state and their underhanded tactics for over a year, had paid our own attorneys for for legal help. They never mentioned how Ancestry had worked a backroom deal with New York State to be the first-ever scanning operation of these files, all just to get a few months’ headway on this public data. And in their publication of the new database, including their press releases and blog posts and even a specially-made video introduction they filmed for social media, they never once mentioned a single word about Reclaim The Records, or our work, or gave a link to our website.

Ancestry will, however, make quite a bit of money off these millions of valuable New York records being posted behind their paywall, an especially important concern as news reports indicate that the company was until recently moving towards an IPO. Their estimated valuation in 2016 was $2.6 billion; Reclaim The Records’ actual valuation in 2016, at the time of our initial FOIL request, was literally zero dollars. We did it for the love of open public records access.

Was all this legal? Sure. We still had our copies of the data we’d won. Ancestry’s copies of the images, being the original ones created directly from the state’s films, technically didn’t come from us, so no credit was required. But was it right to cut all of our work and initiative out of the story? Was it ethical, or even just plain nice?

We will leave this question as an exercise for the reader.

 

Part XI: We share our toys

Meanwhile, Reclaim The Records took our hard-won copies of the statewide death index and returned our copies to the public domain, totally free, just as planned.

We uploaded everything to the Internet Archive, where people can flip through the pages of each year’s films for free, without any paywalls or logins. Most of the files are in good to excellent quality, except for a few years from the 1940’s, where even the state’s vault copies had deteriorated to a terrible degree, showing how critical it was that an organization like us had forced New York State to finally make a digital copy and preserve their own data.

We’ve also reached out to other non-profit genealogical organizations, including to regional groups like the New York Genealogical and Biographical Society, and the New England Historic Genealogical Society, giving them an early heads up about the impending release of the records and asking them if they’d like to host copies on their own websites. Both groups have enthusiastically agreed, and will add the material to their existing data collections in the near future. We’re also thrilled to hear that other genealogical organizations like FamilySearch and MyHeritage will be making their own transcription projects for these images. Eventually there will be free copies of a text database of the New York State death index, not behind paywalls, for everyone to use.

 

Part XII: We FOIL Albany, Buffalo, and Yonkers

There are still a few loose ends to wrap up. The statewide death index was missing deaths for three cities for several decades; Albany, Buffalo, and Yonkers did not cooperate with the statewide registration requirements until about 1915, and their deaths are mostly missing from the index prior to then.

So Reclaim The Records filed three brand new FOIL requests with the city clerks’ offices for these three cities, asking for copies of their death indices prior to 1915. The acquisition and publication of that missing data may take another year to complete, depending on whether or not we will need to initiate legal action against one, some, or all of three of the agencies for non-compliance with the law.

You can read all the back-and-forth discussion with the three agencies here, at the MuckRock pages for our new requests: AlbanyBuffaloYonkers.  (The Buffalo e-mails make for especially fun reading.)

 

Part XIII: MORE AWESOME RECORDS REAL SOON

So that’s the story of the New York State death index, and how we won it. Phew!

Record Set Details:

  • Record Content Type: Death Records
  • Record Years: June 1880 through December 31, 1956
  • Record Format: Index Only
  • Record Physical Format: Microfiche (not microfilm)
  • Number of Records (Estimated): Millions.