FOILing Excessive Processing Fees 1

Thanks to BetaNYC and Civic Hall for organizing yesterday’s Beta Talk on Freedom of Information

Yesterday, I had the opportunity to lead a discussion at Civic Hall about Freedom of Information and the challenges that FOIA Mapper aims to solve. Thanks to everyone who came for the great feedback!

The discussion also took a detour into some of my experience using New York’s Freedom of Information Law (FOIL, the NY State version of the federal FOIA), and also some of the challenges I’ve encountered when requesting government databases.

Database requests (as opposed to requests for physical documents) are an area of Freedom of Information that comes with its own challenges, which are not often discussed. If you’ve ever had a database request stonewalled by excessive fees or an agency’s lack of technical understanding, here are some thoughts you may find helpful.


Database requests / excessive fee estimates

Fee estimates are a tricky aspect of FOIA. Often, before processing a request, agencies will issue an estimated cost. And you are given only two options: agree to pay the fees or withdraw the request.

At times, particularly when requesting a large amount of information in electronic / data format. fee estimates can been exceedingly high. For all intents and purposes, asking a requester to pay $10,000 is a de facto denial. What makes fee estimates tricky is that unlike an explicit denial, a fee estimate is just that, an estimate (not an official agency determination), so you cannot ask the agency for an appeal.

A few years ago, I was looking for information about the potential for solar energy in New York City. I came across the NYC Solar Map on the CUNY (City University of New York) website. Since CUNY is a public entity, subject to Freedom of Information, I submitted a FOIL request for the database that the map pulls from. Below is CUNY’s response.


$300,000 for a copy of the database? To whom should I make out the check?

solar map freedom of information request nyc

Most databases I work with can be copied by pressing the “export” button. Some databases are a bit trickier. But I am not aware of any database that is so difficult to copy, it would require three well-paid engineers working full time for a year.

In fairness to CUNY’s Freedom of Information office, this response came from the team that oversees the map, not the Record Access Officer. I spoke to him after receiving this response, and he actually ended up being pretty helpful, given the circumstances.


When an agency quotes an excessive fee estimate, are they just trying to discourage the request?

$300,000 is easily the most excessive fee estimate I’ve ever received, but when requesting government databases, it is not so uncommon for the agency to respond with an estimate in the $1,000’s or $10,000’s.

I have no doubt that in some cases, fees like this are exaggerated and intended to discourage the request. But in most cases, the reason is due to a misunderstanding, one which is pretty understandable.

It is rare that an agency employee would work with a database directly, or even be aware that there is a database. In most cases, the person in charge of keeping the records does so through a front-end system. And that front-end system may only allow the user to access the records one-by-one. So from the perspective of that employee, “copying the database” means going through each and every record and printing each one out manually.

To see what I mean, take for example this very website.

FOIA Mapper is built on WordPress, which means the post you’re reading right now was pulled from a database. Because you are accessing the website through a front-end system, a web browser, you have no way of knowing what happens on the back-end to serve up the page. If someone were to ask you to copy all 14,000 pages that are on this website, you would have to navigate to each page and copy them one-by-one, which might take quite a while.



The post you’re reading right now (bottom), as well as every other page on this site, is stored in a database

wordpress database foiamapper

Even for someone who maintains a WordPress website, it is not obvious that everything is stored in a single database. But in fact it is. And if you know how to access it, every page on the site can be exported to a spreadsheet with the push of a button.

The situation is similar with many government databases. The person in charge of the system may only be familiar with the front-end interface. And to the best of their understanding, copying the “database” means physically copying the records one-by-one.

Yet, there may be someone in the IT team, someone who may know little about how the system is used, who is able to access the underlying database and export the whole thing in a matter of minutes. In this regard, far from requiring hundreds of hours of work, requests for databases can be just about the easiest requests for agencies to fulfill.


What to do about an excessive fee estimate



If you request a copy of a database and the agency wants to charge you by the page, you can be confident there has been a misunderstanding

foia request database fee

So, you’ve made a FOIA request for a government database, and the agency has come back and asked you to approve a $10,000 fee or withdraw your request.

What now?

I find myself in this situation regularly. And in most cases, I’m able to convince the agency to bring down the fee to a more reasonable number, usually $0. Here’s how.

  1. Before anything, call the FOIA officer who sent the fee estimate. Explain politely that on the back-end of the system, there is likely a database where all the information is stored. And that someone who is familiar with the inner workings of the system can likely export it with the push of a button. Ask them if they could please speak to someone from their IT team and see if this is possible. Follow up with the call with an email, summarizing what was discussed.
  2. If that doesn’t work or if you can’t reach them by phone, send an email explaining the situation (everything above), and ask if you can speak to a technical contact from the IT team who knows this system. Then discuss with the IT contact the data you’re looking for and how it can be extracted. It’s unlikely the FOIA officer you’ve been speaking with has a technical background (although some do), and the communication may be getting lost in translation in both directions.
  3. If the agency does not allow you to speak to a technical contact and has not responded to your questions regarding why the database cannot be easily exported, it’s time to find out why. At this point, I will usually submit additional FOIA requests to get at the nature of the system. Depending on the particular circumstances, here some things you may want to request:
    • Documentation of the record system or the database
    • If the system was built by an external contractor, you can ask for the contract that was signed or the RFP (request for proposal) documents
    • A screenshot of what the user sees when using the front-end the system (how the information is entered, how it is extracted, etc)
    • The agency’s FOIA log, which may contain similar requests (you can check for the FOIA log on the agency’s page here)
    • The emails discussing your original request. Be specific: “I request all emails sent or received by [FOIA OFFICER’S NAME] since [MM/DD/YY] containing either [YOUR FIRST NAME], [YOUR LAST NAME], [THE FOIA REQUEST ID], [OTHER KEY WORDS FROM YOUR REQUEST] or any nouns or pronouns referring to one of the aforementioned items. If you redact any portion of these emails, at a minimum, I want to know where in the emails the aforementioned items appear.”

After this 3rd step, one of two things will usually happen. Either the agency will send you the requested information, in which case, you should be able to submit a new, more specific, request, which leaves less room for the agency to assign a high fee. Or, more likely, the agency will become more forthcoming about what the issue is, realizing it would be easier to work with you than to prolong the situation, dealing with the hassle of a bunch of new requests.


Appreciation for FOIA officers

When you receive an email asking for $300,000 to copy a database, or for that matter, any seemingly unreasonable response to a FOIA request, it’s easy to get angry at the FOIA officer. But in my experience that anger is usually misplaced.


Many internal dynamics play a role in how an agency responds to FOIA requests

foia mapper explained

FOIA officers are generally not the ones determining how much work / expense is involved in processing a request. For that, they have to go to the team who works with the records, and who likely see FOIA requests as an annoyance that takes them away from their job (if you know someone who’s worked in a government agency, particularly early in their career, ask them how much they enjoyed processing FOIA requests).

FOIA officers are in the unenviable position of managing a process with annoyed people on both sides. When trying to navigate a difficult request, it’s important to keep in mind the agency you’re dealing with is not a single entity. There are many people, groups, politics, and internal dynamics that play a role in how the agency responds to your FOIA request.

So, when you receive a FOIA response that seems unreasonable, firing back an angry email (usually my first instinct) may not be the most productive response. In response to high fees, one FOIA requester went so far as to launch an initiative to overwhelm the agency with frivolous requests, apparently to annoy / harass them into behaving more reasonably. Needless to say, I would not recommend this approach either.

In my experience, the best think to do first is speak with the FOIA officer, politely and respectfully, and trying to work with him/her cooperatively to identify the root of the issue.

The “Freedom Summer Initiative” would not be my recommended approach

my two census freedom of information


Resources – database requests / New York City FOIL requests

For New York FOIL requests, a great resource is the Committee on Open Government. On their website, you can find:

The OGIS FOIA Ombudsman is the federal-level equivalent of New York’s Committee on Open Government.

Muckrock is a great resource in many respects. They have an extensive archive of New York City Freedom of Information requests, each of which shows the full exchange of correspondence and ultimately any records that were released. In some cases, you can also find FOIL email contacts that are not published on the agencies’ websites.

DataBridge data dictionary – one particular document, provided by NYC’s MODA (Mayor’s Office of Data Analytics), which I’ve found very helpful in requesting New York City public databases. It is a data dictionary for the City’s DataBridge system, which brings together many databases from various NYC agencies.

Public Advocate de Blasio’s FOIL Scorecard. About a year prior to becoming Mayor of New York City, Public Advocate de Blasio published this report, which assigns each New York City agency a letter grade based on the timeliness of their responses to Freedom of Information requests. The report is helpful for getting a sense of which NYC agencies are the most/least FOIL-friendly. The original announcement is still up on the domain, but the document itself is no longer available.


Public Advocate de Blasio vs Mayor de Blasio – FOIA responsiveness is easier said than done

agency foil grades nyc mayors office


View the full talk I gave on FOIA and FOIA Mapper at Civic Hall


Follow Me

Max Galka

I am an NYC-based entrepreneur and an adjunct lecturer at UPenn. I'm fascinated by data visualization and the ways that data is transforming our understanding of the world. I spend a lot of time with my face buried in Excel, and when I find something interesting I write about it at Metrocosm / Huffpo / Guardian Cities.
More about my background
Follow Me

Leave a comment

Your email address will not be published. Required fields are marked *

One thought on “FOILing Excessive Processing Fees