Documentation · page 7 of 7

Where the data comes from

This page says where each part of StartupGraph comes from, how a fact is checked before we call it verified, which sites we do not read and why, and what is not right yet. It describes the code that answers the API today. The matching rules in full are on Contributing facts, and how much is on file is on What the data covers.

Three ways in

• Imported. Importers wrote most of the list from public registers, lists and sites. Each company names its importer in import_source. A field whose value came from an importer reads status: "imported" and verified: false in fields on GET /api/companies/{slug}: no check of ours stands behind it. A funding round works the same way: each round has verified and status on GET /api/companies/{slug}, /funding, GET /api/funding-rounds and the MCP tools. status is one of verified, unverified, imported: verified when a verified funding round fact stands behind the round (it put the round on file, or it states the same company, day and type), unverified when a contributed fact put it on file and has not verified, and imported when an importer wrote it and no fact has checked it. verified is true only for verified. An imported round's source_url is the link the importer stored, which our verifier has not read (#571).
• Contributed. Anyone with a key can send a fact with a link to a public page (POST /api/facts, MCP submit_fact) or a new company (POST /api/companies). The fact is checked against that page, as below.
• Our own data crank. StartupGraph's own key sends facts through the same door as anyone, and they are checked the same way. Strikes never revoke it. Instead it is paused, with a 429 quality_pause, when more than 10% of its newest 200 decided facts were leads or rejected, or more than 2% were answered with a server error, once at least 50 are decided.

Asked with added_since or sort=added, GET /api/companies says which of the three added each company in added_by: imported, contributed or system.

Lists as sources

Some companies came from a Wikipedia list, not from a page about the company. Each imported field then cites the list it came from (#568):

• From the list of unicorn startup companies: List of unicorn startup companies.
• From a list of one acquirer's acquisitions: that acquirer's list, chosen by acquired_by. Where we have no record of which list, source_url is null and source says the list was not recorded.
• From Wikipedia categories, lists of defunct companies and other lists, where the importer took the company's name from a page title: that page.

Before this, every Wikipedia row linked to a page built from the company's name, which often did not exist or was about something else. The stored rows did not change; the answer is built differently.

Importers we stopped

Some sources' terms forbid what we do with their data. We stopped the importers that read them (#557). Nothing schedules them, their commands refuse to run and say why, and a job already queued ends without fetching.

Source	Why it is stopped
TechCrunch (funding rounds, news, discovery)	Personal, non-commercial use only, and no automated copying.
Y Combinator	Its terms forbid scraping and commercial use.
Crunchbase (discovery)	No storing or redistributing its data without written consent.
Wellfound	Its terms forbid harvesting and commercial use.
Product Hunt	Its API "must not be used for commercial purposes".
OpenCorporates	Free keys are for journalists, NGOs and academics, and the data is share-alike (ODbL).
LinkedIn (headcounts)	The User Agreement forbids scraping.

For Crunchbase only the discovery importer is stopped. Importing a Crunchbase export file that a person supplies still runs, and is under review (#560).

The rows these importers wrote earlier are still on file and still served. Replacing those values from sources we can use is open (#260), and so is whether to keep the funding rounds read from TechCrunch (#465). The Hacker News importer is not stopped. The GitHub trending importer no longer reads the trending page; it asks the GitHub search API for the most-starred new repositories.

How a contributed fact is checked

• It is served at once, marked unverified. A fact changes the company as soon as it arrives, with verified: false, and is checked in the background. Whether to hold every fact back until it is verified is open (#363, #364).
• We fetch the page and look for the claim. The check is mechanical: the company's name and the value, in the forms people write them. Contributing facts lists what it matches and where it fails.
• Some sites verify on their own. A claim found on one of these is verified: find-and-update.company-information.service.gov.uk (Companies House), ycombinator.com, techcrunch.com, ft.com, theinformation.com, prnewswire.com, fortune.com, forbes.com, cnbc.com, theverge.com and wired.com. So is one on the company's own website, under the rules below.
• Any other site is held for a second source (needs_second_source). See Held claims.
• A page that answers 404 or 410 does not exist. The fact is rejected and the key gets a strike. Three strikes revoke the key. A rejected fact's value is taken back out of the company. Cleaning up facts rejected before that rule is open (#241).
• A page we cannot open is tried again, every hour. After 6 tries over at least 24 hours it reads source_unreachable: kept as a lead, not served, no strike.
• A page that does not state the claim, or needs a sign-in, makes the fact a lead. A sign-in page is never fetched.
• Verified is one thing. There is a single verified flag. It does not yet say whether the source was the company's own word, one accepted page, or two people agreeing (#277).

A company's own website

The website we trust for a company is accepted on its own for facts about that company. Never a website that is still being claimed. These rules decide what counts as the company's own site:

• A listing, profile or platform site is never anyone's own site (#559). That covers profile and directory sites (crunchbase.com, wellfound.com, producthunt.com, pitchbook.com and others), app stores (apps.apple.com, play.google.com), hosts where anyone can get a page (github.io, vercel.app, sites.google.com, linktr.ee), social sites, and parked domains. The host is matched in full, so sites.google.com is never anyone's own site while google.com is still Google's. If a company's website on file is a page on one of these hosts, no other page there counts as the company's word: a stranger's page on crunchbase.com that names the company and the claim is held for a second source.
• Such a website is refused when it is sent (#561). A website or investor_website fact, or a tip, on one of these hosts answers 422 website_not_own_site. Nothing is stored and it does not count against the key's daily limit. A short, fixed list in the code covers the few companies that are one of these sites, such as Y Combinator on ycombinator.com and GitLab on gitlab.com. Nothing a contributor sends can add to that list.
• A homepage can confirm the address by a short name (#559). A website claim cited to the company's homepage verifies when the page names the company by its short name ("Pano" on pano.ai for Pano AI), as long as a trusted website on that site is already on file and the page we land on is on it too.
• A site does not vouch for itself. A website claim for a company with no trusted website, cited to a homepage that names it, is held for a second source. So is a fund's website cited to the fund's own homepage.

Held claims

A claim we could not verify yet is held, not dropped, in one of three states:

• needs_second_source: the page states the claim, but the site is not accepted on its own. It is served, marked unverified. It is verified when a different person sends the same claim citing a different site, or when its sender sends it again citing an accepted site that states it. Two keys on one email are one person, and two pages on one site are one site.
• Lead: the page did not state the claim, needed a sign-in, or could not be opened. It is private and not served. We fetch the page again: the waits between checks are 1, 3, 7 days, and the last wait repeats, until 60 days after it was sent. A sign-in page is never fetched again. Every newly verified fact is also compared with the open leads.
• source_unreachable: a lead whose page we could not open after the tries above.

The sender can give a held claim a better page by sending the same claim again with a different link. We check it against the fact on file right away. At most 3 new pages are checked for one fact, each once; a page we could not open can be sent again after 24 hours. When a fact takes a new page, the page it cited before is kept in our records and is not served.

Sites that refuse us

These sites refuse every request our verifier makes: Business Wire (businesswire.com), GlobeNewswire (globenewswire.com), Reuters (reuters.com), Axios (axios.com), Bloomberg (bloomberg.com), The Wall Street Journal (wsj.com) and the SEC (sec.gov). So the verifier never fetches them. A fact citing one reads source_unreachable on its first check. That is not a rejection: the refusal says nothing about the claim, so there is no strike. None of them is on the list of sites accepted on its own. Sent again as a better page for a held claim, one of them is answered at once and does not use up one of those pages. To get such a claim verified, cite another page that states it, such as the company's own announcement.

robots.txt and terms

• A page a person cites. The verifier fetches the one page a contributor pointed us to. It does not read that site's robots.txt first (#465).
• Pages the API fetches on its own. The API has two passes that fetch pages nobody pointed it to: looking for an open tip's claim on pages we already know for that company, and re-checking a trusted website's home page. Before each fetch, and each redirect, the code reads the site's robots.txt and goes ahead only if it allows the page for * and for each of ClaudeBot, Claude-Web, anthropic-ai, Claude-User and Claude-SearchBot. A robots.txt that answers 404 allows everything. Any other failure counts as no.
• Sites those two passes never fetch, whatever their robots.txt says: techcrunch.com (its robots.txt), prnewswire.com (its terms), fortune.com (its terms), cnbc.com (its robots.txt) and forbes.com (its robots.txt), plus every site that refuses us.
• Job boards and GitHub are read by the API through their public APIs: Greenhouse, Lever and Ashby for open roles, the GitHub API for stars.
• The importers are not covered by these rules. The importers that still run read public APIs and files: companies-house, crunchbase (a file a person supplies), edgar, github, github-orgs, wikipedia, wikipedia-acquisitions, wikipedia-categories and wikipedia-defunct, plus Hacker News discovery. None of them checks robots.txt or the lists above, and the EDGAR importer reads sec.gov, which refuses the verifier. This is open (#576).
• The data crank fetches pages through its own process, outside the API. This page does not describe its rules.

So for TechCrunch: a TechCrunch page a person cites still verifies a fact, our importer no longer reads TechCrunch, and the API's own fetching never goes there.

What is not right yet

• Some headcounts were invented. A seeder in the importer toolchain made up a headcount history for companies, and a current headcount for those with none, from a hash of the company's name. Removing those values is open (#256). synthetic_history on /headcount is that seed data, not evidence; snapshots carry a source.
• Facts are served before they are verified (#363, #364), and verified does not yet say how (#277).
• Values from the stopped importers are still served (#260, #465), and the Crunchbase file import still runs (#560).
• The importers do not check robots.txt, and the EDGAR importer reads sec.gov (#576).

Corrections

To correct a value, send it as a fact with a public page that states it (Contributing facts). It is checked like any other. If you think we got a check wrong, write to hello@startupgraph.dev.

Check it yourself

• GET /api/companies/{slug}: fields gives each value with its source_url, source, verified and status.
• GET /api/companies/{slug}/facts: the contributed facts on file that you may read, with their status.
• GET /api/changes: every fact applied, verified or rejected, in order.
• GET /api/coverage and GET /api/pulse: how much is on file and what landed recently.

Every call is in the reference.