Clinical Evidence Intelligence

A dermatology review can involve hundreds of papers from several databases. The same paper may appear more than once. Full text may be missing. Titles and abstracts use inconsistent terms. Public patient discussions add useful context but cannot be treated as clinical evidence.

Doing collection, screening, tagging and synthesis by hand was too slow for repeated research across eczema, acne, psoriasis and topical steroid withdrawal.

I built a five-stage Python pipeline for collection, deduplication, screening, analysis and report preparation. Each output stays linked to its source. Academic evidence and public patient reports remain separate throughout the process.

1. Run defined Boolean searches against Semantic Scholar and PubMed. Parse PubMed XML and collect titles, abstracts, authors, publication details, citation counts and open-access links.
2. Collect public Reddit posts through a separate route for patient-reported experience. These records are labelled as anecdotal data and never merged with trial evidence.
3. Use fuzzy title and author matching to remove duplicate papers returned by different services.
4. Look for open-access full text through Unpaywall, Europe PMC and CORE. Use browser automation for accessible download pages and store files in authorised Google Drive storage.
5. Run a first screening pass that scores relevance from 0 to 2 against a written rubric. Low-scoring papers are excluded with the reason recorded.
6. Extract text from retained PDFs with PyMuPDF. A second pass records condition, symptoms, intervention category, treatment theme, evidence strength and reported effect in a fixed schema.
7. Run a similar but separate screen over public patient reports to record common concerns and treatment experiences.
8. Convert each paper into a numeric text representation called an embedding. UMAP reduces the number of dimensions so the data is easier to process. HDBSCAN then groups papers with similar text without requiring a fixed number of groups in advance.
9. Calculate a silhouette score to measure how clearly the groups are separated and flag weak clusters for review.
10. Write the screened records and labels to Airtable, then build thematic tables across condition, intervention and evidence strength.
11. Generate draft treatment summaries only from the screened dataset. Every claim remains tied to the retained evidence and requires human review.

Preparing and screening a 500-paper review manually would take 45 to 60 hours. Automated collection, deduplication and first-pass screening reduce the work to roughly eight to twelve hours of reviewer time.

That is about an 80% time saving. Every retained paper has a source record, inclusion reason and structured evidence fields, which makes the final review easier to audit.