Some Ways I Think About Evaluating Grant Applications

where I'm most likely to disagree with others

Rider-Waite Tarot, 6 of Pentacles

I’ve done enough grant evaluations so far (for ACX grants and SFF) and been involved in philanthropy in various other contexts, at work and informally, that I have developed some idea of how my opinions and intuitions differ from other people’s.

I thought it might be interesting to share some of my “tastes”. Not everybody has to have the same tastes or funding philosophy, but these are mine.

#1: It’s The Donor’s Money

In my worldview, charitable donation is optional. Generally praiseworthy, but optional.

And the purpose of donation is to buy outcomes that the donor wants to see in the world. You donate to make the world more like the one you want to live in. Generally, a reasonable person’s values go beyond strictly personal consumption; one also cares about what kind of a society one lives in, what other people’s lives are like, what sorts of institutions exist, what sorts of things humanity has created or discovered, and so on.

As an agent doing research or evaluation on behalf of a donor, I try to find opportunities that fit in the intersection between my own values and the donor’s. If there weren’t substantial overlap in our values, it would be silly for them to take my advice in the first place.

Sounds obvious. But it is sometimes a point of disagreement, because I’m not using a utilitarian framework where there’s a single ranking of the goodness of causes. “Should” donors give to political causes, for instance? Well, it depends on the donor’s worldview — how useful do they think political action is, what kind, etc. I see it as inappropriate for the “peanut gallery” of public observers to claim the authority to tell donors what their values should be.¹

#2: Importance, Neglectedness, Tractability

I think the old familiar “ITN” framework is pretty solid, and works even if you’re not using a utilitarian framework.

• importance = good done / % of a problem solved

• tractability = % of a problem solved / % increase in resources

• neglectedness = % increase in resources / extra person or dollar.

I don’t generally see much value in quantifying these numerically, since “amount of goodness” is often judged intuitively and there’s no principled way to put all kinds of goodness on a single numerical scale. But it’s still useful verbally when arguing the merits of one project relative to another.

To make it more “gutfeel friendly”, I generally think of it in a cruder way, without the percents.

• Importance: how much would I (or the donor) care if this project succeeded fully at its goal? How big a deal is this?

• Tractability: how likely do I think this project is to succeed at its goal? (Or, how far do I expect them to get?)

• Neglectedness: how much does this donation make the difference between the project happening and not happening?

#3: Yay Community Infrastructure

I tend to like proposals for “community infrastructure” projects, like coworking spaces and events, that serve a community that the donor is part of.

Somewhat controversially, I even like donations for “community infrastructure” that serves a social, aesthetic, or recreational purpose. (For instance, I think it’s good for opera-lovers to donate to their local opera.) Donation is buying outcomes; it’s good to invest in, and support, things that you and the people around you will enjoy. That’s how we get and keep nice things. It’s a form of responsible self-interest, which is good by my lights; you’re not passively hoping someone else will support the things you enjoy, you’re doing your part to contribute.

There’s a neglectedness angle too; nobody but members of a community has any reason to donate to community infrastructure. So “if not us, then who?” is an argument in favor of prioritizing “taking care of our own”.

Finally, I like community infrastructure because it’s often concrete, and thus harder to fake. If you’re operating a physical space or running an in-person event, people can come and evaluate it directly. It forces contact with reality in a way that doesn’t happen by default with theoretical research and writing. I place a lot of value on “doing a real thing” compared to doing a potentially-bigger but potentially-unreal thing.

#4: Yay Niche Topics

Sometimes, there’s a topic that’s only really interesting to a group that includes the donor and applicant.

For example, there’s a very small niche of people who know and care that psychedelics seem to treat cluster headaches. Psychedelics people, cluster headache sufferers, and people who are really interested in qualia, are much more interested in this topic than anybody else.

Unlike community infrastructure, the topic might objectively matter for many people outside the community, but maybe it’s not salient or familiar to them. (How many people are passionate about far-UVC disinfection?)

So that’s a neglectedness consideration, just like for community infrastructure; “if not us, then who?”

#5: Yay “Technical” Work

I have a bias towards projects that accomplish something “technical”, in the sense of demonstrating unfakeable, rare skill at structured processes.

Building something physical is “technical” in this sense. Experimental lab science is “technical”. So are math (and associated theoretical fields) and software to some extent, though they are becoming less technical with the rise of AI assistance.² So are arts and crafts that are “technically challenging.”

It’s a little hard to explain why.

Some of it is that I respect skill, want to reward it, and think skill is a component of human flourishing; it makes some sense to me to fund the humans who are best at various things to keep doing what they’re excellent at.

Some of it is that skill is a resource that I’d like to direct to good ends.

Some of it is that skill (and effort) is a sort of indirect signal of caring weighted by credibility. The fact that a capable person invested a lot into a project means that it’s more likely that someone I have reason to respect, strongly believed it was worth doing, so maybe I should believe it’s worth doing too. It’s the opposite of a “YOLO” application where someone just thought they’d spend a minute to ask for money in case they get it, with no particular conviction or commitment. An investment of skill and effort is a bet that something is worth further investment, and that counts for something.

#6: Yay Straightforward Public Information Resources

Sometimes a project is just making a public organized compilation of information to answer a straightforward question.

Like, “What’s everyone doing in AI safety research?”

Or, “How many people die from every disease, worldwide?”

I almost always like these projects. They’re (often) cheaper to make in the age of AI, so the budget I’d allocate to each one is smaller than it used to be, but I still want lots more of such projects to exist.

The utility of an information resource is pretty obvious and not super contingent — information is valuable in a lot of potential world scenarios. Information resources also don’t take a ton of specialized expertise to evaluate, and it’s unlikely for them to fail to meet their goals. It’s easy to be confident these projects aren’t “fake”; they are what they say on the tin.

#7: Yay “Cool Shit”

I have a soft spot in my heart for projects that simply “glorify the human spirit”. Things like pure discovery or exploration (of space, nature, archaeology, etc) or artistic works that aim at genuine beauty, grandeur, or awe.

Not everything with this intention is effective, of course; quality matters; but in general I think these sorts of projects are underrated.

Any time you’d say “man, wouldn’t it be cool to” and somebody goes and does it, that falls in this category.

The Herculaneum Scroll Prize is a prime example of “cool shit” philanthropy.

#8: Two Cheers for Meta

I used to dismiss “meta” projects like fellowship programs, workshops, grant programs, etc. I thought “why not directly fund people who do the research itself? This managerial meta layer is make-work.”

I first got convinced otherwise by seeing the success of Laura Deming’s Longevity Fund. I remember thinking, “why would this brilliant young scientist start a VC fund instead of, y’know, doing science herself?” And then I grew to appreciate the (obvious) answer: because there was no money for the science, and no money for translating the science into biotech companies and taking it into clinical trials. It was genuinely too early to “just do the thing directly”. “Doing the thing” depended on institutions and communities and resources that didn’t yet exist. There’s much more money in aging/longevity now, and the Longevity Fund deserves a lot of credit for paving the way.

Another example I’ve come to appreciate recently is the success of research fellowship programs like MATS. A lot of good AI alignment research (in my opinion) has come from MATS graduates. There was a time in my memory when basically nobody with serious math/CS/machine learning chops was interested in AI safety/alignment/etc. (For that matter, there was a time in my memory when almost nobody in math/CS/ML seemed to be interested in AI, as in the mind-like nature of certain algorithmic processes, at all! Yann LeCun’s students were outliers there!) Training and talent development can make a big difference to a nascent research field.

In general, making a new or neglected research field “a thing”, making it comprehensible and attractive to strong researchers, via money, communication, convening, institution-building, etc, is valuable work.

Like all coordinative or managerial work, field-building can be make-work, and its value is indirect and hard to assess from the outside, but you’ll have a hard time getting things done if you don’t do any.³

I do tend to think field-building is most important where there’s very little of it. Fields that basically don’t exist outside a few underfunded individuals really need more coordination, communication, training, and fundraising. Established fields, not so much.

I also think it usually doesn’t make sense for me-as-grant-evaluator to recommend another grant-disbursing organization, unless it has specialized expertise or does things besides grantmaking. If you have more money than time, it can make sense to delegate grantmaking; if I have plenty of time and (obviously) agree with myself more than I’d agree with anybody else, I have no reason to delegate to another generalist.

#9: Gumption Counts

I would occasionally see someone apply for a grant to build a project around scaling and applying some innovation, and see the critique that it’s not novel, because some academics had also published about that method/technique.

My counterpoint was usually that, well, this person applied for a grant from us, to do the thing full time and at scale. That shows gumption: the person is entrepreneurial enough to seek out unusual sources of funding, committed enough to the project that they want to take it farther and devote more of their time to it, and visionary enough to believe in and articulate how it can be brought into the real world.

Just because other people have published papers on related topics doesn’t mean those authors are a.) still active in the field, b.) interested in using funding to pursue the idea further, or c.) energetic/ambitious/decisive enough to lead a new effort that Gets The Job Done. If you have time, it may be worth consulting the people who’ve done related work to see what they think of the applicant. But the applicant is the one in the arena, and that counts for something, all else equal.

#10: Yay Personal Relationships

Of course, it’s important to disclose conflicts of interest, and of course it’s healthy to avoid being too cliquish/insidery or only funding the “usual suspects” who are already well-networked in a social or professional scene.

But I personally don’t feel too many qualms about giving “bonus points” to projects where I already know and think well of the applicant. “I want to fund my friends” is bias. But “I know this person to be trustworthy” or “I know how they think and I like their style” is actually very strong evidence in someone’s favor, compared to what you can learn about someone totally unfamiliar, from only an application and public information, in a short time. There’s a reason why so many institutions tend to lean on personal networks! And while you do want to fight the temptation to bias or cliquishness, it would be just as wrong not to put a thumb on the scale for Known Good people.

Sometimes I think people are trying too hard to assume a View from Nowhere, and are uncomfortable admitting that they are particular people, situated in a particular place with a particular perspective and set of social relationships, and that this is fine. You can only do your best from where you are.

#11: Check For Ideological Orientation

One thing I wish more grant evaluators would do is to look up the people involved in a project, especially a project that has bearing on politics or society, and see what their political or ideological orientation is. Also, see who the other funders are, and what their orientation is. And then think about whether you want that.

This doesn’t mean you should be ideologically doctrinaire or refuse to fund anybody you disagree with about anything. Diversity of opinion is healthy. And paranoia about guilt by association is a fast train to Crazytown.

But. Look people up. Look up what they were doing before this project. Be familiar with what keywords are indicative of what worldviews. If there are belief systems you consider to be Bad News, you don’t want to fund them unintentionally.

#12: Filter Slop Aggressively

Low effort applications — AI-generated, or just good old fashioned manual halfassery — get more common the more visible your grant opportunity is and the less gatekeeping around it. I like having open applications and being open to being surprised by good ideas from anywhere, but the flip side of that is that you have to be pretty ruthless about not spending too much time deliberating over the low-quality, low-effort stuff.

I believe in using Pangram, for instance. And unceremoniously throwing incomplete applications in the trash.

I also believe in trying to quickly verify independently that the applicant is a real person with a LinkedIn or other professional profile that is at least vaguely in the right vicinity for being the sort of person who’d be qualified for a grant. You don’t want to be unfair to people with nontraditional backgrounds, but you do need to use some quick-and-dirty heuristics like “for a research grant, almost all qualified applicants will have a PhD and a job at a research institution, or be a student pursuing same.” Because people do fabricate identities, and people do apply for things they’re wildly unqualified for!

#13: Yay Outcomes

This one isn’t very controversial, but I have occasionally encountered disagreement: it’s a mark in a project’s favor when they can list concrete, legible accomplishments.

Some bullet points of what they’ve done, links to places I can read more or (especially) see the output of their work, are much better than not having those things.

I’m also a big fan, where possible, of giving the first dollar to “outsiders” who aren’t well networked or polished or working within prestigious institutions. But it’s a lot easier to gain confidence in those people if they can show results. Ideally, a results orientation and an open mind are complements, not opposites.

#14: Why Donate Rather Than Invest?

If an organization is a for-profit company, I think there needs to be a specific reason why funding should be a donation rather than an investment. Subsidizing unsuccessful businesses is a bad idea in general.

I do think there’s a place for investment that takes into account the impact or benefit of an organization as well as its financial ROI. When I make (very small) angel investments, I prioritize things that I think make the world a better place, and certainly don’t invest in things that I think make the world a worse place. I think it’s normal to care about both the state of the world and financial returns, and it’s kind of dysfunctional that some investors insist on only making investments based on maximizing financial returns while artificially disregarding all their other values.

But I do think there should be a defeasible presumption against donating to for-profits. Usually, for a for-profit company with a mission, success at the mission should correlate positively with business success. Also, if it’s possible for you to capture financial upside from your funding, why wouldn’t you want to?

There are exceptions, as when a donation would allow a company to do something that doesn’t maximize profits but does good, in a way they couldn’t otherwise do.

It can get complicated, and I’m not always sure of my opinions on this issue. But I am, by default, inclined to not favor donations to perfectly normal businesses that should be able to stand on their own two feet.

1. Of course it’s fine to personally think someone has values contrary to your own and is donating to a bad cause, and it’s fine to say so, but you shouldn’t seriously expect them to care based on your say-so. Persuading someone to change their fundamental priorities is a big project that requires trust, intimacy, and mutual respect.

2. “Less technical” doesn’t necessarily mean “less good.” AI-generated proofs may still be valid (and interesting) and AI-generated software may still be functional (and useful). But it’s less of a costly signal of (rare, structured, process-based) human skill. Good automated work is more of a sign that the humans involved are good “generalists”, in the sense of having good judgment or good ideas or good priorities. That’s also a lot harder to evaluate and harder to understand; and it takes less effort to make a few good choices than to grind at a “technical” task, so it’s less of an indication of conviction.

3. I’m in the field-building business myself these days, so I would say this, but I do actually believe it is valuable. Yes, it’s a tremendous privilege to be able to live in the “world of ideas” supporting researchers, but I also think we people in “support roles” contribute to causing good research to happen.