198 lines
4.2 KiB
Markdown
198 lines
4.2 KiB
Markdown
# Phase: Ingest — Process a New Source
|
|
|
|
Run when user says "ingest this", "add this article", "process raw/<file>", "ingest https://...".
|
|
|
|
## Step 1 — Get the source
|
|
|
|
**If URL provided:**
|
|
```bash
|
|
firecrawl scrape "<url>" -o raw/<slug>.md
|
|
```
|
|
Slug = domain + path slugified. Example: `ai-2027.com-main.md`
|
|
|
|
**If file already in raw/:** proceed directly.
|
|
|
|
**If text pasted:** write it to `raw/<slug>.md` first.
|
|
|
|
## Step 2 — Read existing index
|
|
|
|
```bash
|
|
# Check what's already in the wiki
|
|
Read wiki/index.md
|
|
```
|
|
|
|
Scan index for pages that might be related to this source. Note their paths.
|
|
|
|
## Step 3 — Read the source
|
|
|
|
```bash
|
|
Read raw/<slug>.md
|
|
```
|
|
|
|
For large files (>200 lines), read in chunks. Use `grep` to find key sections first.
|
|
|
|
## Step 4 — Discuss (optional but recommended)
|
|
|
|
For important sources, briefly discuss with the user:
|
|
- 2-3 most important takeaways
|
|
- Any surprises or contradictions with existing wiki content
|
|
- Which existing pages are most affected
|
|
|
|
Skip this for batch ingestion.
|
|
|
|
## Step 5 — Write summary page
|
|
|
|
Write `wiki/sources/<slug>.md` using this structure:
|
|
|
|
```markdown
|
|
---
|
|
title: "Source Title"
|
|
date: YYYY-MM-DD
|
|
type: source
|
|
tags: [tag1, tag2, tag3]
|
|
source-url: https://...
|
|
source-type: article | paper | video | transcript | book | podcast
|
|
ingested: YYYY-MM-DD
|
|
---
|
|
|
|
# Source Title
|
|
|
|
> One-sentence summary of the core claim or finding.
|
|
|
|
## Key Points
|
|
|
|
- Most important point, specific and concrete
|
|
- Second key point
|
|
- Third key point
|
|
(3-7 points, each a complete thought)
|
|
|
|
## Details
|
|
|
|
Expanded explanation of the most important ideas. 2-4 paragraphs.
|
|
Write as synthesis, not transcript. Future-you should be able to read
|
|
this instead of the original source.
|
|
|
|
## Key Quotes
|
|
|
|
> "The most memorable direct quote from the source."
|
|
|
|
## Implications
|
|
|
|
What this means for the field / for your work / for related topics.
|
|
|
|
## Questions Raised
|
|
|
|
- Open question this source raises
|
|
- Another unresolved question
|
|
|
|
## Related
|
|
|
|
- [[Concept Page Name]] — how it relates
|
|
- [[Entity Page Name]] — connection
|
|
- [[Another Source]] — agrees/disagrees because X
|
|
```
|
|
|
|
## Step 6 — Update entity and concept pages
|
|
|
|
For each person, organization, concept, or project prominently mentioned:
|
|
|
|
**If page exists:** append new information under a dated section:
|
|
```markdown
|
|
## [YYYY-MM-DD] New Finding from [[Source Title]]
|
|
- New data point or updated claim
|
|
```
|
|
|
|
**If page doesn't exist:** create it:
|
|
|
|
Entity page (`wiki/entities/<slug>.md`):
|
|
```markdown
|
|
---
|
|
title: "Entity Name"
|
|
type: entity
|
|
entity-type: person | org | product | place
|
|
tags: []
|
|
first-mentioned: YYYY-MM-DD
|
|
source-count: 1
|
|
---
|
|
|
|
# Entity Name
|
|
|
|
> One-sentence description.
|
|
|
|
## Background
|
|
...
|
|
|
|
## Key Contributions / Role
|
|
...
|
|
|
|
## Appearances in Wiki
|
|
- [[Source where first mentioned]] — context
|
|
```
|
|
|
|
Concept page (`wiki/concepts/<slug>.md`):
|
|
```markdown
|
|
---
|
|
title: "Concept Name"
|
|
type: concept
|
|
tags: []
|
|
first-mentioned: YYYY-MM-DD
|
|
source-count: 1
|
|
---
|
|
|
|
# Concept Name
|
|
|
|
> Definition in one sentence.
|
|
|
|
## Explanation
|
|
...
|
|
|
|
## How It Works
|
|
...
|
|
|
|
## Examples
|
|
...
|
|
|
|
## Connections
|
|
- [[Related Concept]] — relationship
|
|
- [[Opposing Concept]] — how they differ
|
|
|
|
## Sources
|
|
- [[Source Title]] — what it says about this concept
|
|
```
|
|
|
|
**Limit:** touch at most 15 pages per ingest to keep sessions focused.
|
|
|
|
## Step 7 — Update wiki/index.md
|
|
|
|
Add a row in the appropriate section:
|
|
|
|
```markdown
|
|
| [[sources/source-slug\|Source Title]] | YYYY-MM-DD | source | One-line summary |
|
|
```
|
|
|
|
## Step 8 — Update wiki/overview.md
|
|
|
|
If the source significantly changes the high-level picture, revise the relevant section of `wiki/overview.md`. Add a note: `> Updated YYYY-MM-DD after ingesting [[Source Title]]`
|
|
|
|
## Step 9 — Append to log
|
|
|
|
```markdown
|
|
## [YYYY-MM-DD] ingest | Source Title
|
|
|
|
- Source: `raw/<slug>.md`
|
|
- Summary page: `wiki/sources/<slug>.md`
|
|
- Pages updated: [[Page A]], [[Page B]], [[Page C]]
|
|
- New pages created: [[New Entity]], [[New Concept]]
|
|
- Key takeaway: One sentence.
|
|
```
|
|
|
|
## Quality Checklist
|
|
|
|
- [ ] Summary page written at `wiki/sources/<slug>.md`
|
|
- [ ] At least 2 existing pages updated with new information
|
|
- [ ] New entity/concept pages created for major new subjects
|
|
- [ ] `[[WikiLinks]]` used throughout (Obsidian-compatible)
|
|
- [ ] `wiki/index.md` updated
|
|
- [ ] `wiki/log.md` appended
|
|
- [ ] No raw source files modified
|