Data methodology
How Subalytics data is collected, transcribed and matched
What happens between a podcast episode or post being published and a record appearing in the API: sources, transcription, speakers, company matching and the fields to use for point-in-time research.
featured in
Data teams need to know how a dataset is built before they trust it. This page covers collection, transcription, speaker identification, company matching, corrections, point-in-time fields and join keys. Field names match the API reference.
Collection
Subalytics collects publicly available content and profile data from each platform: published podcast episodes from tracked shows, public Substack newsletters and posts, public Bluesky accounts and posts, and public Medium publications. Podcasts are checked daily, Bluesky posts arrive in near real time, and creator profiles are captured as weekly snapshots.
Public sources and provenance
- Only public content is collected. For paid Substack posts, only the publicly visible portion is collected.
- Every record links to the original episode or post, with the time it was published and the time Subalytics first found it, so any data point can be checked against its source.
- Creators can ask to have their profile removed: see the opt-out page.
- Data is licensed per product line: creator data, podcast data and company mentions.
Transcription
New episodes from tracked podcasts are transcribed every day in a run that starts at 05:00 UTC. Episodes usually appear in the API within a few hours. Each transcript is split into segments with start and end times in seconds from the start of the audio.
Only episodes published since a podcast was first tracked are transcribed in the standard feed. A transcript can be produced again later; when that happens its updated_at changes.
Speaker identification
Each segment has a speaker label that points to the transcript's speaker list. Labels are per transcript: SPEAKER_00 in one episode is unrelated to SPEAKER_00 in another.
For each speaker, Subalytics tries to identify a name, a role in the conversation (host, co-host, guest), a title and a company. Identification is automatic and best-effort, and any of the four can be empty. Names can be filled in after a transcript is first published, which updates its updated_at. A segment's speaker can be missing where voices couldn't be separated.
Company matching
Subalytics keeps a list of monitored public companies with ticker, CIK, exchange, sector, industry and known aliases and products. New podcast transcripts, Substack posts and Bluesky posts are checked against that list.
| Match type | Example | Notes |
|---|---|---|
cashtag | $TSLA | Explicit ticker with $ |
company_name | Tesla | The company's name |
alias | Alphabet → Google | Known alternative names |
product | instagram → META | A product owned by the company |
ticker | TSLA | Ticker without $ |
Confidence
- Every mention has a confidence score between 0.5 and 1.0.
- Mention endpoints default to 0.7 and above; you can set min_confidence lower or higher.
- Daily and weekly counts in entity-mention-stats include only mentions at 0.7 or above.
- Podcast mention endpoints return the passage with the mention and its timestamps, not the whole transcript.
Corrections and withdrawn mentions
Matching improves over time and past content is re-checked. That can add mentions, change their confidence or remove a mention that turned out to be wrong. A post whose last mention is removed disappears from the content endpoints.
Additions and changes show up when you sync with updated_from. Removals don't, so if you need an exact match with the API, re-pull a recent window, for example the last 30 days, on a schedule.
Point-in-time fields
Use availability time, not publish time, in backtests
A podcast published on Monday and transcribed on Tuesday was knowable on Tuesday. Filter on transcribed_at or created_at to keep backtests to what was available at the time, and store updated_at so you can tell when a record changed after you first saw it.
| Field | Where | Meaning |
|---|---|---|
published_utc | Episodes, posts, mentions | When the podcast or writer published the content |
created_at | Episodes, transcripts, mentions, posts | When Subalytics first found or produced the record |
transcribed_at | Episodes | When the transcript became available |
updated_at | All records | When the record last changed (re-transcription, speaker names, re-scoring) |
transcribed_from / transcribed_to | Episode filters | Select episodes by when they became available |
updated_from / updated_to | Most endpoints | Select records by when they last changed, for incremental sync |
Join keys
Records are linked by stable Subalytics IDs, so data from different endpoints joins on the same keys. Only entity-companies carries CIK, exchange, sector and industry; other entity endpoints carry symbol and company name, so join on symbol for the rest.
| Key | Identifies | Joins |
|---|---|---|
symbol | Company ticker | All entity endpoints; your security master |
cik | SEC Central Index Key | entity-companies; filings and other SEC data |
sub_id | Creator or podcast | Mentions ↔ creator and podcast data (audience, history) |
episode_sub_id | Podcast episode | Episodes ↔ transcripts ↔ podcast mentions |
mention_id | One mention | Deduplication and tracking corrections |
content_id | The post or episode | Mentions ↔ the content endpoints |
Creator metrics
Creator data is captured as weekly snapshots. Subscriber and follower counts come from what each platform shows publicly; Substack free and paid counts are recorded separately where the publication shows them. Counts can differ from a creator's own claims, which may include audiences on other platforms.
Limits
- No sentiment scores. Each mention comes with its passage instead.
- Speaker identification is automatic and best-effort.
- Transcripts cover episodes since tracking began, unless historical data is delivered separately.
FAQ
Common questions
Filter on availability fields (transcribed_at for transcripts, created_at for mentions) instead of published_utc, and store updated_at to see when a record changed after you first pulled it.
Yes. Past content is re-checked as matching improves, so mentions can be added, re-scored or removed. Re-pull a recent window on a schedule if you need an exact match.
Use symbol (ticker) for all entity endpoints and cik from entity-companies for a permanent identifier.
Contact us
Have a question or a custom use case?
Tell us what you're trying to monitor or search across, and we'll get back to you within a business day.
Questions from your data team?
We'll walk through the fields, timestamps and matching with your engineers and researchers.