Transparent Censorship: An In-Depth Look at FOIA in 2015 1


Which government agencies receive the most Freedom of Information requests? How many of those requests yield responsive documents? And of those that do not, what are the most commonly stated reasons?

Perhaps most importantly, does the Obama administration live up to its claim of being the “most transparent” administration ever? (Spoiler Alert: No)

Last month, federal government agencies released their 2015 Annual FOIA Reports, which include detailed stats on how they responded to each and every Freedom of Information request.

Here is a rundown of what they reported.

 

What the AP found

The only analysis of the 2015 FOIA reporting that I’ve found so far comes from the Associated Press, posted shortly after the reports were released. Here is what they concluded.

The Obama administration set a record for the number of times its federal employees told disappointed citizens, journalists and others that despite searching they couldn’t find a single page requested under the Freedom of Information Act

foia requests granted obama
There is no doubt the AP’s conclusion is factually correct. The government failed to find the requested records 129,825 times last year, a new high. And a time series plot of the percentage of requests granted (shown in this chart) shows an alarmingly steep downward trend.

In 2008, the last year of the Bush administration, 41% of FOIA requests were granted in full. Since Obama took office, that number has steadily declined, hitting a new low of just 23% in 2015.

Having now spent a few days digging through the data myself, I think the AP gets the big picture right: under this administration, the government has become less transparent

However, contrary to what the high-level figures imply, agencies seem to be turning up as many records as they always have. As explained below, what’s changed is the likelihood that those records arrive with portions erased or withheld.

To accomplish this new level of censorship, agencies have expanded their use of one particular, broadly-defined clause in the Freedom of Information Act: Exemption 7(e).

Building on what the AP reported, here is my take on the data.

 

A handful of agencies accounted for the majority of FOIA requests

  • In total, federal agencies processed a record-high 769,133 Freedom of Information requests last year.
  • The top 5 agencies handled 71% of all FOIA requests.
  • The Department of Homeland Security Alone handled 45% of all requests.

Here is how the requests broke down by agency in 2015. Each square is sized relative to the number of requests processed by that agency.

foia requests processed by agency

 

The top 5 agencies: the Dept of Homeland Security (DHS)the Dept of Justice (DOJ), the Dept of Defense (DoD), the Dept of Health & Human Services (HHS), and the Dept of Veterans Affairs (VA) together accounted for 71% of all Freedom of Information requests.

Of those, the busiest agency by far was the Department of Homeland Security. Last year, it processed 348,878 Freedom of Information requests, about 45% of the government-wide total.

 

How federal agencies responded to FOIA requests, 2008-2015

  • Though the government responded with “no records found” a record-high 129,825 times last year, when viewed in proportion to the number of requests that were made, that figure is more-or-less in line with previous years.
  • Agencies are granting requests at the same rate as in 2008, but it has become more common for the documents to arrive with pieces withheld or removed (a “partial grant”).

The chart below shows how agency responses to FOIA requests have changed over time.

If you separate the requests into two categories, those that yielded documents and those that did not, here’s how it shakes out.

 

Requests that returned documents: ~60% of cases

The two bottom areas of the chart represent cases in which the agency responded with something (more below on what that something could be). Overall, the total has not changed much over time, but the balance has shifted away from full grants (a complete set of documents) toward more partial grants (documents with portions withheld or erased).

  • The percentage of requests yielding full grants has fallen from 42% in 2008 to 23% in 2015.
  • Partial grant responses have risen from 18% in 2008 to 40% in 2015.

 

Requests that returned nothing: ~40% of cases

The four brown-yellow sections in the chart are those requests where the agency came back with nothing, for one reason or another. Likewise, this total has also remained fairly constant over time. And so have each of the four individual components. Here’s a rough breakdown.

  • 15% of the time, agencies find no responsive documents.
  • 5% of the time, they do find documents, but withhold them entirely as legally exempt from disclosure.
  • 3% of the time, the request is withdrawn by the requester (read: got scared away after the agency quoted a high processing fee).
  • The remaining 17% includes overly broad requests, duplicate requests, referrals to another agency, and other miscellaneous outcomes.

 

Not more denials, more partial denials

As shown above, FOIA denial rates and “no records found” rates have remained constant since 2008. What has increased is the frequency of partial grants (which you could just as easily call partial denials). The trend is clear when you look at the stats in aggregate, and also when you look at each agency individually.

The table below shows the percentage of requests that received partial grants in 2008 and in 2015. Together, the 20 agencies shown accounted for 97% of all FOIA requests that were processed last year. And all but 3 of them are awarding more partial grants today than they were in 2008.

Partial grants by agency (% of FOIA responses)

Top 20 AgenciesPartial Grants in 2008Partial Grants in 2015Change
Dept of Homeland Security51.3%62.2%+10.8%
Dept of Justice10.6%14.5%+3.9%
Dept of Defense20.8%23.4%+2.7%
Health & Human Services1.2%13.6%+12.4%
Veterans Administration3.4%31.8%+28.4%
National Archives1.4%5.3%+3.9%
Social Security Admin.0.9%24.0%+23.1%
Dept of Agriculture*9.8%11.4%+1.7%
Equal Employment Opportunity Com.*66.8%70.5%+3.7%
Dept of Labor37.0%43.7%+6.7%
Securities & Exchange Com.2.9%3.4%+0.5%
Dept of State16.2%16.3%+0.1%
U.S. Treasury13.4%23.0%+9.6%
Dept of Transportation25.1%22.2%-2.9%
Environmental Protection Agency4.6%6.5%+1.9%
Office of Personnel Mgmt1.4%26.7%+25.3%
Dept of Interior21.3%26.3%+5%
National Labor Relations Board*8.6%21.4%+12.9%
CIA*29.5%29.1%-0.4%
U.S. Postal Service*20.4%19.1%-1.3%

 

redacted foia response

A “partial grant” from the Bureau of Prisons (Muckrock’s redaction hall of shame)

What does it mean that agencies are awarding more partial grants?

It’s hard to say. A partial grant could be as trivial as the removal of a few inconsequential words. At the other extreme it could be a bunch of entirely blacked-out pages. It could also be the withholding of entire documents.

For an entertaining look at some of the most egregious examples of “partial grants”, check out Muckrock’s redaction hall of shame.

There is nothing in the annual FOIA reports about how much information agencies are withholding from partial grants. But the reports do cover agencies’ use of FOIA exemptions, which give some clue as to the type of information that is being withheld.

 

Use of FOIA exemptions: what kind of information are agencies withholding?

  • In 2015, the government applied FOIA disclosure exemptions 829,058 times.
  • 2015 was the first year in which the number of exemptions used (829,058) exceeded the number of requests that were processed (769,133). This is possible because a document may have several pieces withheld, and each piece may be justified by more than one exemption.
  • The most commonly used exemptions were Ex. 6 & Ex. 7(c) (invasions of personal privacy) and Ex. 7(e) (information that would reveal law enforcement investigation techniques).

Agencies are only permitted to withhold / redact information if it falls under one of 14 legal exemptions (technically 9 exemptions, one of which has 6 separate sub-exemptions).

The chart below shows how many times each exemption was applied in 2015 as compared to 2008.

The most common justifications for withholding information: exemptions 6, 7(c), and 7(e)

foia exemptions

Exemptions 6 and 7(c): “unwarranted invasions of privacy”

The two most commonly used exemptions, Ex. 6 and Ex. 7(c), both deal with unwarranted invasions of personal privacy (e.g. medical information, home phone numbers, etc). The difference between the exemptions is subtle. Ex. 7(c) protects against “unwarranted invasions of privacy” from the release of law enforcement records. Ex. 6 applies to all records, not just law enforcement records. And it protects against “clearly unwarranted invasions of privacy,” a higher bar than Ex. 7(c).

I’m sure other people would disagree, but in my experience these two exemptions are fairly specific. And the information that gets withheld is normally not even material. For example, when requesting an agency’s FOIA logs, I don’t think anyone really cares whether the document includes the requesters’ home addresses.

 

Exemption 7(e): “techniques and procedures for law enforcement investigations”

The third most commonly used exemption, Ex 7(e), is more controversial. Here is how it’s defined:

Exemption (7)(E) protects from disclosure information which would reveal techniques and procedures for law enforcement investigations or prosecutions or that would disclose guidelines for law enforcement investigations or prosecutions if disclosure of the information could reasonably be expected to risk circumvention of the law.

Most of the other exemptions do not allow agencies to withhold information unless they can point to a specific negative consequence that would result from public disclosure. In contrast, Ex. 7(e) does not require the government to show a particular determination of harm or risk of circumvention of law, giving agencies much more discretion over what they can withhold.

Also, the term “law enforcement” has a more general meaning that you might initially think, covering the enforcement of both criminal and civil law. Here are a few examples of what it includes:

  • Tax evasion
  • Immigration enforcement
  • Terrorist threats
  • Emergency response procedures
  • Aviation watch lists
  • Financial crimes
  • Domestic surveillance
  • U.S. customs

 

To read more about the different FOIA exemptions, the FOIA Advocates website gives a good summary.

 

Is the government becoming more or less transparent?

Since 2008, agencies have become increasingly aggressive with their censorship of documents. It has now become the norm for FOIA-ed documents to arrive with portions removed or erased. This trend is true of the government as a whole and of almost every agency individually.

What kind of information is being withheld today that wasn’t being withheld in 2008? Based on the growing use of Exemption 7(e), it appears agencies have decided to become less transparent about their activity behind the scenes to enforce the law.

If the biggest question of today’s government is whether greater safety is worth sacrificing freedom, the data gives a pretty good idea of which direction the the current administration is leaning.

foia full grants vs partial grants

 

More

If you’d like to dig through it yourself, here is the raw data from the 2008-2015 Annual FOIA Reports:

You can find the annual reports themselves on the DOJ website.

If any part of the reporting data is not clear, here is the the government’s Handbook for Agency Annual FOIA Reports, which does a good job of documenting it all.

There are for sure many ways of interpreting the data, but if there is any doubt where the current administration stands on transparency, here are some recent stories that answer the question pretty clearly.

 

Let me know if you have a different opinion.

 

 

 

Follow Me

Max Galka

I am an NYC-based entrepreneur and an adjunct lecturer at UPenn. I'm fascinated by data visualization and the ways that data is transforming our understanding of the world. I spend a lot of time with my face buried in Excel, and when I find something interesting I write about it at Metrocosm / Huffpo / Guardian Cities.
More about my background
Follow Me

Leave a comment

Your email address will not be published. Required fields are marked *

One thought on “Transparent Censorship: An In-Depth Look at FOIA in 2015

  • Heloise

    “the most transparent administration ever”… just a political tagline. Nothing surprising about that, every politician spouts a bunch of rhetoric and conceals their real objectives.