On October 14, 2020, Reclaim The Records made a Freedom of Information Act (FOIA) request to the United States National Archives and Records Administration (NARA), asking for billions (yes, billions) of digital images and their associated text metadata, to return online access to American historical documents to the public.

This is the full text of that FOIA request, which we submitted online through the MuckRock platform:

FOIA Request for all digital images and text metadata created through NARA’s public-private digitization partnership program

To Whom It May Concern:

This is a request under the Freedom of Information Act.

I represent a 501(c)(3) non-profit organization called Reclaim The Records. We are an activist group of genealogists, historians, journalists, teachers, and open government advocates. We acquire genealogical and historical databases and images from government sources, including government archives, often through the use of Freedom of Information laws. We then upload those records to the Internet, without any copyright or usage restrictions or paywalls, making them freely available to the public and returning these taxpayer-funded materials to the public domain.

 

PART I: BACKGROUND FOR THIS REQUEST

The United States National Archives and Records Administration (NARA) has for several years managed an innovative public-private partnership program to digitize many of the important historical documents they hold, particularly records that would be useful for family history research. These include multiple enumerations of the United States Federal Census (through 1940), immigration and naturalization records, military and veteran records, tax assessment lists, and more.

More than four hundred of these important historical record sets have been digitized so far under this long-running partnership program, with each of those record sets containing hundreds of thousands, or more often millions, of individual documents. A likely-incomplete listing of these record sets is available on the NARA web page “Microfilm Publications and Original Records Digitized by Our Digitization Partners” located at https://www.archives.gov/digitization/digitized-by-partners . The total number of unique historical documents digitized and transcribed through this program is probably in the billions.

In exchange for having private corporations and non-profit organizations agree to become “partners” and digitize these historical records from their original paper or microfilm formats — a massive task that would be largely cost-prohibitive for NARA to conduct on its own — NARA agreed to let these partners have the exclusive use of those newly-digitized materials on their own websites for a certain amount of time, an “embargo period”.

This grant of a supposedly exclusive entitlement to public records was meant to induce these partners to spend their time and money to conduct the records digitization and transcription at their own expense, instead of at the taxpayer’s expense. But while well-intentioned, it also meant that these original historical records were often completely removed from public access while the companies worked on them, making the records functionally unavailable to researchers, sometimes for years.

And even once the digitization and transcription work was finally completed, the exclusivity period for each newly-created digital record set was also supposed to be time-limited. After the stated embargo period would end for each unique record set, usually within five years but sometimes in three years, NARA would then be able to freely disseminate the now-digitized versions of these public documents, both the images and the text metadata that accompanied them. NARA’s own policies state that the agency could and would publish the digital copies through NARA’s own website or in their official online Catalog or through their official API access or through other means. See item number two from “NARA Principles for Partnerships to Digitize Archival Materials” at https://www.archives.gov/digitization/principles.html :

“2. After an agreed-upon period of time, otherwise known as an embargo period, NARA gains unrestricted rights to the digital copies and the associated metadata transmitted to NARA by the partner, including the right to give or sell digital copies in whole or part to other entities, if NARA so chooses. If resources permit, we will try to make the digital materials available in our online catalog within the same year they are no longer in the embargo period.”

But in practice, this simply hasn’t happened. NARA has never actually posted online the vast majority of these records that were digitized through their partnership program, not to their Catalog nor indeed anywhere else where the public might be able to freely access and download the now-digital records. This remains the case today, even when the embargo periods for many of these record sets have been expired for more than a decade, sometimes two decades. A small number of the records are now finally online in the NARA Catalog, but even there, the data sets are still not available to the general public as bulk image or bulk data downloads and are cumbersome to search or use individually.

Instead, literally billions of these historical American records remain solely in the hands of NARA’s primary digitization program partner, Ancestry.com. Ancestry is a private corporation, previously co-owned by a private equity firm and the government of Singapore’s sovereign wealth fund, until they were sold to a different private equity firm for $4.7 billion in August 2020. Ancestry has purchased several smaller companies in the genealogy and family history space over the past few years, including the companies Fold3.com and Archives.com, both of which had previously independently been included in NARA’s digitization partnership program. Thus, the vast majority of the billions of records digitized through NARA’s partnership program are now available only behind Ancestry’s subscription paywall, or through companies now owned by Ancestry with their own additional subscription paywalls. Annual subscriptions to these websites can cost hundreds of dollars per year per person.

NARA surely did not mean to create a de facto monopoly on nearly all digital copies of important American historical documents like the Census and immigration records and military files, all for the benefit of a single private corporation. But by not making the no-longer-embargoed documents available to the public anywhere else, not even on NARA’s own website, and leaving them solely in the hands of their mostly-commercial partner organizations, that is exactly what has happened.

NARA’s own “Principles for Partnerships to Digitize Archival Materials”, as referenced above, clearly states in item number seven that:

“Public access to publicly owned resources will remain free. Partners may develop and charge for value-added features, but access to the digital copies ultimately should be readily accessible and free…NARA will have unrestricted ownership of these copies, including the right to make these copies freely available online for download.”

However, in practice, NARA has also repeatedly denied independent requests for copies of even subsets of this voluminous partnership-created digital data. We are aware of at least three different entities, two genealogy-related corporations and one non-profit organization, none of which were NARA digitization partners, who each independently requested and were each denied access to copies of this data through e-mails, phone calls, meetings, and other discussions with NARA leadership. In all three cases, NARA denied the requests, saying that NARA would put the records online themselves, through their Catalog or API…eventually.

Thus, the end result of NARA’s digitization partnership program has been that billions of important American historical documents were successfully digitized and transcribed — but then were mostly not made available to the public for decades in any way other than by requiring the public to buy expensive annual data subscriptions benefiting private corporations, primarily a single multi-billion-dollar conglomerate, whose previous owners included a foreign government.

We at Reclaim The Records would now like to make an official request for open public access to these important American historical records.

 

PART II: OUR REQUEST

Under the Freedom of Information Act, we at Reclaim The Records request copies of the following:

1) We request every single record created under NARA’s public-private digitization partnership with the entities Ancestry.com, Fold3.com (formerly known as Footnote, now owned by Ancestry), Archives.com (now owned by Ancestry), and FamilySearch (a non-profit organization). We do not request any records that were created through NARA’s partnership with other smaller entities, such as the Daughters of the American Revolution (the DAR). Specifically:

1a) We request all of the digital images, in their original, full-size, uncompressed, and non-watermarked versions.

1b) We request all of the associated text metadata (names, dates, places, etc.) also created under the partnership agreement, which goes along with those images, making them searchable. For example, a spreadsheet or database may have been created for each data set that lists the name of each person referenced in each image, along with the date, the location, or other extracted information such as place of birth, marital status, volume number, census enumeration district, microfilm reel number, or any other text information relevant to that particular data set and/or each individual image.

1c) We request all copies of finding aids, training materials, handbooks, checklists, formatting guidelines, data dictionaries, data templates, data lists, or other internal documentation that explains more about the digitization of these images and the transcription and compilation of their associated text metadata, and how they relate to each individual data set.

2) We also request any records that were digitized under NARA’s partnership program that may not have been properly delivered or returned to NARA after their digitization was completed. We have heard stories about records that remain solely in the possession of certain partner corporations, for which NARA never collected the files upon completion of the image scanning and the text metadata entry. We therefore request copies of all the partnership-created digital images, associated text metadata, and finding aids (or data dictionaries, documentation, templates, etc.) for those previously-undelivered files, as well. To be clear, we contend that NARA is required to collect these records from these companies and produce them to us in response to our request and we are requesting that NARA do so.

 

PART III: FORMAT OF PRODUCTION

We request that all of these files, the images and text metadata and finding aids and data dictionaries and so on, be turned over to us in their original digital formats, as they were delivered to NARA by the partners, or turned over for the first time if the partner never delivered the final files to NARA as they should have.

We would like to receive our copies of this information on portable USB drives. We are willing to pay the costs for purchasing those drives and for their insured and trackable domestic shipping. However, we believe some of this data may already be stored online in the Amazon Web Services (AWS) S3 Glacier system, which we believe NARA uses for its internal file storage. If this is the case, then for any data sets that are already completely online in AWS S3, we would consider receiving just the online versions of those specific data sets, by having that data copied directly from NARA’s AWS S3 bucket(s) into Reclaim The Records’ AWS S3 bucket(s), and those data sets would then not need to be downloaded to a USB drive.

Please inform us of all fees in advance of fulfilling our order.

 

PART IV: REQUEST FOR FEE WAIVER

We also request to be treated as a “media requester” for the purposes of calculating the fees for this FOIA request. We are a non-profit organization, not a commercial entity. We do not charge for copies of any of the tens of millions of records we have already acquired from government agencies and released to the public. We are one of the largest open records organizations in the United States. As of October 1, 2020, our e-mail newsletter, which has been published several times a year for the past six years, now has a circulation of over 7,500 subscribers. Our social media outlets such as our Facebook page have more than 11,000 followers, and our Twitter account has more than 6,100 followers.

We have even created several free standalone websites to both disseminate and discuss the data that we receive from government entities. As just one example, please see our website https://www.MissouriDeathIndex.com/ and our multiple associated newsletter issues linked from that website. We don’t just release data sets, we discuss them too, using our editorial skills and discretion, and then disseminate those discussions to our readers.

Therefore, under 45 CFR 1602.2, we believe that we properly meet the legal qualifications as a “media requester” entity, and so we would need to pay only any duplication fees after the first 100 pages of material, and we should not need to pay any search fees or review fees.

Thank you for your consideration, and we look forward to your timely response within twenty business days, as the statute requires.

sincerely yours,

Brooke Schreier Ganz, on behalf of Reclaim The Records
info@reclaimtherecords.org
https://www.reclaimtherecords.org/

We will provide updates when we learn how NARA chooses to respond to our request.

In September 2017, Reclaim The Records made a Freedom of Information Act (FOIA) request for a copy of what is colloquially called the BIRLS File from the United States Department of Veterans Affairs. This is a Veterans Benefits Administration database that provides information on deceased individuals including name, birth and death dates, branch of service, and enlistment and release dates. This request was one of the first two FOIA requests our organization had ever made for a nationwide database under the federal FOIA, rather than a state-level law such as a state Sunshine Law.

Our original request was filed through the MuckRock website:

To Whom It May Concern:

This is a request under the Freedom of Information Act.

I represent a 501(c)(3) non-profit organization called Reclaim The Records. We are an activist group of genealogists, historians, journalists, and open government advocates who acquire genealogical and archival data sets and images from government sources, often through the use of Freedom of Information laws. We then upload those records to the Internet, making them freely available to the public and returning them to the public domain.

We request a copy of the Beneficiary Identification Records Locator Subsystem (BIRLS) Death File, a very large database maintained by your agency. This data has already been made available to the public before, and it is currently online at at least two major commercial genealogy websites, Ancestry.com and Fold3.com. We are now requesting a copy for our organization, so that we may post it online for free public use.

We would also like to recommend that your agency upload a copy of the file to the federal government’s open data portal, data.gov.

Based on this scanned “Request for Records Disposition Authority” document, located at archives.gov…

https://www.archives.gov/files/records-mgmt/rcs/schedules/departments/department-of-veterans-affairs/rg-0015/n1-015-11-002_sf115.pdf

…it seems that the BIRLS Death File has the VA Item Number 08-065 000, and a copy may have already been turned over to the National Archives and Records Administration (NARA).

We request that this file be turned over to us in database format, as it already exists in a database. We would like to receive it on a portable USB drive, and we are willing to pay the costs for purchasing that drive and for its insured and trackable domestic shipping, as well as any fees associated with fulfilling this request. Please inform us of all fees in advance of fulfilling our order.

Thank you for your consideration, and we look forward to your timely response within twenty business days, as the statute requires.

The request was originally rejected by the VA for privacy reasons on April 6, 2018 (PDF). But after our attorney contacted the agency, they wrote us a new letter on April 26, 2018 (PDF) where they said they would be referring the issue on for further study in a different department, rather than closing the request.

On July 26, 2018, the VA wrote again, denying the records for the second time (PDF). This time, they claimed to us that the BIRLS database had been “erroneously” been released to Ancestry.com on March 18, 2011 pursuant to FOIA request number 10-02722-F, and that they didn’t want to release the data to us, too. They also claimed that this BIRLS database release amounted to “a data breach of Veteran’s personal information” — which seems rather unlikely given that everyone in the database is, by definition, dead.

And so, on August 6, 2018 we appealed the second denial (PDF), for much the same reasons as we had the first denial. We did not get a response.

On September 17, 2018, Reclaim The Records sued the US Department of Veterans Affairs in US District Court (SDNY) for the BIRLS Database. The case is still pending.

Then three weeks later, on October 10, 2018, the VA suddenly reached out to our attorney by e-mail with a new response letter. The letter said that the VA had suddenly “remanded back” the case to their FOIA officer for “further consideration, appropriate processing, and the issuance of a subsequent IAD” — and this “oops, our bad!” letter was very conveniently dated September 13, four days before the FOIA lawsuit was filed. ?

In mid-2020, we won our lawsuit, in mid-2022 we finally received the files from the VA, and in late 2024, we sent this newsletter out to our subscribers explaining the rest of the story:

our fiftieth! the big five-oh! newsletter

NEW DATABASE OF EIGHTEEN MILLION US VETERAN RECORDS NOW ONLINE, WITH A BUILT-IN FREE-FOIA-FILING SYSTEM!

search for a name, push a button, and a few months later get a DVD full of scanned files in your mailbox from the VA, ALL FOR FREE

Hello again from Reclaim The Records! We’re that little non-profit which likes to pry historical and genealogical files and databases out of government archives, libraries, and agencies, and then puts them all online for totally free public use. And we’re here in your inbox to announce a really unique and exciting resource that our work has identified and now made available: the first-ever free public access to the BIRLS database, the main index to the VA’s records of veteran benefits files. This particular story has a lot of background to explain, so we’re actually going to have to break up this announcement into a few separate newsletters over the next few weeks.

But the super-short version of the story is this: we figured out how to get access-by-FOIA to some amazing veteran records from the VA, from the late nineteenth century to the present day, really unusual records that aren’t available anywhere else, and now we want to help you get these files, too.

And so we’re going to start at the end of this long story and show you some of the results we’ve gotten in the past few months: the amazing, unique, almost-always-never-seen-before-by-any-researcher historical files you can now get about pretty much any deceased US veteran who served in the late nineteenth or twentieth centuries, whether they’re a relative of yours, a research interest, or just a famous person whose file you’re curious to peek at. A few examples showing the variety of things you can find in these files:

• a 1961 handwritten letter from a WWII veteran and actor named William Lubovsky, instructing the VA to please refer to him by his long-time stage name “Will Lee”, although he would later become even better known as Mr. Hooper on “Sesame Street”.
• a 1919 application for a military life insurance policy for a WWI soldier and immigrant named John (Giovanni) Primo, giving information about his beneficiary — his non-immigrant mother back in Italy, who was likely born in the mid-19th century.
• a 1987 life insurance payout receipt for a WWII Navy veteran upon his death, payable to his wife — the veteran being Broadway and film director Bob Fosse and his long-separated wife listed on the receipt being dancer Gwen Verdon.
• A 1988 handwritten letter from a veteran explaining to the VA with some amusement that the Navy’s offer of “education and training benefits” was quite unnecessary, as she had already received her PhD from Yale in 1930, thankyouverymuch. The veteran was Rear Admiral Grace Murray Hopper, a pioneer in computer programming.
• a 1958 typewritten letter from an undistinguished and poor WWI veteran (and barely-known relative of the person writing this newsletter) facing increasing medical problems, asking for financial help from the VA to supplement his meager pension from the local Cake Bakers Union.
• A 1945 medical and psychological evaluation of a WWII veteran suffering from bilateral trench foot and what we would now call PTSD. It includes a harrowing first-hand account — recounted in the VA doctor’s notes in his file — about his experiences with hand-to-hand fighting in foxholes in France and Germany. The veteran was Jack Kirby, a comic book artist and writer, who just five years earlier had co-created Captain America, and who would go on to co-create other superheroes, including the X-Men, the Avengers, and the Fantastic Four in the 1960s. But lesser known in his career was his artwork on a 1950s war-related comic book series that was literally called “Foxhole”— perhaps unjustly obscure, given Kirby’s military service, as recounted in his full VA benefits file, which wound up being 211 pages long.

And those are just six of the veteran stories that we uncovered, using this new dataset of more than eighteen millionveterans’ names to launch our FOIA requests. It’s more than a century’s worth of data, finally available for free public use.

So here’s the slightly-longer version of the story, and how we did this. A few years ago, we at Reclaim The Records tried to get a copy of a certain large data set called BIRLS from a certain federal government agency, the US Department of Veterans Affairs, better known as the VA. Even thought the VA had already given out an earlier copy of that database (but with fewer years of data) to the commercial genealogy behemoth Ancestry.com a few years before that, the government refused to give the data to us. The VA actually claimed that their own internal data was so terribly messy and so badly curated that they simply couldn’t give it out to anyone, and thus the BIRLS database should remain unavailable to the general public forever — or else solely available in an older and smaller dataset (covering fewer years) that was locked up behind the $300/year paywall of a single commercial genealogy company.

This seemed rather unfair to us, for a government agency to prioritize handing taxpayer-funded data to a single commercial entity, but not to the public. Also, it seemed like a terrible new reasoning for federal government agencies to try to evade their responsibilities under FOIA: just be really sloppy at maintaining your agency’s internal data and then you’ll always have an excuse to not be able to give any of it to the public.

So obviously, we sued the VA in federal court to get the records under the Freedom of Information Act (FOIA), because that’s the sort of thing we do.

And after fighting about it for a few years, we won our lawsuit and thereby won the database! And the data we won was a far more complete version of the BIRLS database, containing all records where the VA believed the veteran had died prior to mid-2020. But we didn’t publicly announce our win at the time, because the judge’s ruling had also stipulated that the VA didn’t have to turn the data over to us until about two more years after the ruling, to give the VA more time to try to clean up their own internal data mess. But eventually they gave us a copy, and it’s finally online now.

We just launched a search engine for that data, a free new mini-website at BIRLS.org. The information in this database covers over eighteen million deceased US veterans, possibly the single largest public data set on US veterans ever released. And this is the first time it’s ever all been free and public (rather than paywalled), and it’s even downloadable, if you for some strange reason prefer to directly download ginormous CSV files.

But the giant database we won is really just the index to the underlying files. And so —this is the really unusual part of our story! — we also built a really cool new feature into this searchable website:

If you find someone of interest listed in the search results at BIRLS.org, you can also send a FOIA request to the VA right from your web browser asking the VA to send you the full underlying benefits claims file from their warehouse for that particular veteran.

These files we’ve helped to shake loose from the VA, the ones which are being indexed by the BIRLS database, are veterans’ benefits claim files, also called C-Files (not to be confused with the immigration-related C-Files you can get from USCIS). These C-Files are a compilation of all the different types of benefits claims a veteran (or their relatives) made (or tried to make) to the VA related to their previous military service: claims for health care, disability or life insurance policies, educational benefits (the GI Bill), mortgage assistance (VA loans), and more.

Basically, these files are the modern versions of the Civil War pension files or other veteran files held at the National Archives (NARA) which genealogists have used for years. Except these files are still held at the VA, because they’re much newer, and ~95% of them have never been sent over to NARA, and so almost nobody has ever seen them or used them.

“Filing the FOIA for them” means the VA will search for and pull the actual folder from their warehouse shelves, digitally scan the contents, and send it to you by mail on a DVD, or occasionally as paper photocopies. There could be ten items in that folder, or fifty, or even hundreds of pages of materials, from medical reports, to letters, to computer punch cards for pension payouts, even copies of vital records like marriages and divorce decrees, and there’s really no way to know what’s in a file in advance. You just have to search, and then make the FOIA request, and then wait a few months to see what you get.

And if you can’t find a deceased veteran’s name listed in the database, don’t worry. The VA may still have a file about them which you can request, but it just might not have been indexed into the BIRLS database. This is especially likely if the veteran (or their family) didn’t have any ongoing contact with the VA in or after the 1970s, which is when the BIRLS database first started getting built. So to deal with that possibility, we also built a new “build your own FOIA” system into our new website, so you can still make FOIA requests for C-Files even if the veteran’s name isn’t indexed in the BIRLS database. (They have a slightly lower chance of working, but hey, they’re free!)

Oh, and there are quite a few non-veterans listed in the BIRLS database too, including some civilians who worked for federal agencies like NOAA, who might have been entitled to some types of benefits such as VA home loans. And there are even a surprisingly large number of non-US veterans included in the BIRLS database, specifically Filipino nationals who served in the Philippine Commonwealth Army, the Philippines Scouts, the Philippine Guerrilla and Combination Service, or other service during the time period that the Philippines was an unincorporated territory and commonwealth of the United States, from 1935 to 1946.

Like we said, there is a lot of background and material to explain with this new data release. And this is just the beginning.

We’ll have a lot more to say about the BIRLS database contents (and the database’s flaws), the multi-year lawsuit we had to file in SDNY to get this material out of the VA and into the public domain, and more of the incredible stories we’ve found when FOIA’ing veterans’ C-Files from the late nineteenth century to the (almost) present day. But for now, we think it’s time to stop writing this first newsletter and turn the website over to you guys, to let you start searching and discovering and requesting records for yourself.

A huge thank you to to the approximately fifty beta testers from both the genealogy and FOIA research communities, who quietly worked with us on this project over the past few months. They gave us suggestions, made bug reports, and collectively filed more than 1,100 FOIA requests for C-Files, as we all stress-tested the system, meaning both our shiny new website and the VA’s creaky internal processes for file production. In the coming days and weeks, some of the beta testers will be posting their own blog entries and social media reports about the great stuff they’ve been finding in these C-Files about their relatives or research interests.

And now that we’re going public with the new website and its FOIA-filing capabilities, we hope that you too will find incredible materials.

Finally, we want to emphasize that we’re bringing this project to the general public for free. We filed the original lawsuit for free, we built the website for free, and we designed and built the new free-FOIA-filing-by-fax (yes, fax!) system for free. But hiring lawyers, running servers, and handling fax and e-mail APIs and data storage costs and so on — well, that’s not actually free to us. And we’re not a commercial genealogy behemoth with an expensive subscription model, we’re a little non-profit.

So if you want to see more great projects like this one, where we employ Feats of Strength to make government agencies turn over incredibly important data sets to the public domain, and where we have the Airing of Grievances to explain how to find, acquire, and use these once-hidden amazing files… Well, ’tis the season to be generous, and our little non-profit would cheerfully accept a Festivus present or two. Your generous support makes projects like this one possible.

We hope you enjoy the new website, and find terrific things in the new files!