Getting a course from Canvas

1. Why it runs inside Canvas

Two limits shape the design:

• Simeon's page can't call Canvas. It is a local file:// page (or getsimeon.com): another origin, with none of the student's Canvas cookies, and Canvas doesn't allow cross-origin reads of its API.
• A file:// page can't message a Chrome extension at all.

So the export runs on Canvas's own origin, in a page where the student is signed in through their school's usual sign-in (SSO, MFA, whatever it is). Every API call is fetch(url, { credentials: 'include' }). Nothing about the session is stored by Simeon: in the apps, the web view's own data store keeps it between launches, as a browser would.

This also sidesteps the usual blocker for Canvas tools. Personal access tokens are often switched off by schools, and an OAuth developer key needs the school's admin. A student's own session needs nobody's permission and can only read what the student could already open.

2. Getting messages in and out

In Chrome

Simeon page ──postMessage──▶ bridge.js ──runtime message──▶ background.js
                                                               │ finds a Canvas tab, or opens
                                                               ▼ one in the background
                                                mapper.js, in the Canvas tab
           ◀── bridge.js ◀── background.js ◀──── events: progress … done | error
                                                zip ──▶ Downloads

• bridge.js is a content script on Simeon's page. Because it is matched on every local file, it does nothing unless the page has <meta name="simeon-app">. It shares the page's window, so the two talk with postMessage.
• background.js (the service worker) finds a Canvas tab or opens one in the background, and pings it every 500 ms for up to about 20 seconds. A tab opened before the extension was installed never got the content script, so after two failed pings on a loaded Canvas tab it injects it with chrome.scripting; mapper.js guards itself so it never runs twice. If Canvas never answers, it's nearly always the school's sign-in page, so that tab is brought to the front.
• mapper.js answers an export at once ({ ok, started }) and then reports through separate event messages addressed to the tab that asked. Nothing waits on one long-lived message: a course with its files takes minutes, longer than an MV3 service worker stays awake, and each event wakes it again.

In the apps

The apps have no extension, so a small canvas-bridge.js stands in for the two things a content script expects:

1. chrome.runtime. A fake sendMessage posts events to the app; the app asks mapper.js things by calling its listener exactly as Chrome would.
2. A browser that saves <a download>. mapper.js hands over the finished zip by clicking a download link to a blob. The bridge wraps URL.createObjectURL to remember the blobs it made and HTMLAnchorElement.click to catch a click on one of those, then sends the bytes to the app as base64 in 3 MB slices, which the app reassembles and writes atomically. Every other click on the page behaves as usual.

On the Mac the scripts run in an isolated content world, so none of these overrides are visible to Canvas's own scripts. The web view re-injects after each navigation that lands back on the Canvas host (a new page is a new JavaScript world), opens sign-in pop-ups ("Sign in with Microsoft") in the same view, and is told not to throttle itself when off screen, so the export carries on at full speed once its sheet closes.

Windows needed one more piece. A Canvas file download redirects to Amazon S3, and WebView2 (Chromium) blocks the page from reading that cross-origin response, which Chrome lets the extension do and the Mac's WebKit doesn't enforce. So the Windows bridge wraps fetch for same-host GET …/files/:id/download only, and the app fetches the file itself: it sends the window's Canvas cookies to the Canvas host only, follows the HTTPS redirects without them, and streams the bytes back in 6 MB pieces. Status codes and the rate-limit header are passed through, so the back-off below still works.

3. Being polite: cfetch

Every request goes through one function:

• At most three in flight across the whole export. A freed slot is handed straight to the next waiter.
• Backs off when told to. A 429, or a 403 whose body says "rate limit", waits 1, 2, 4, 8, 16 seconds (capped at 30) and tries again, up to five times.
• Watches its budget. When X-Rate-Limit-Remaining drops below 150 it pauses a second; below 50, three.
• Read-only. Every call is a GET. Nothing on Canvas is ever posted, submitted or changed.

Lists are fetched with per_page=100, following the Link: <…>; rel="next" header to the end. A page that isn't OK ends the list quietly, so an endpoint a course has switched off gives an empty list, not a failed export.

4. What it reads

First the course, with include[]=syllabus_body, term and teachers. Then, three on the wire at a time:

Pages when the Pages tab is off. Many instructors hide the Pages tab, and the list 404s, but each page can still be read by its slug. The export takes the union of the list and every module item of type Page, and fetches each.

Discussions. A topic listed with an empty body is fetched again on its own. Every topic's replies come from /discussion_topics/:id/view, which returns the whole reply tree and its participants in one call.

5. Finding every file

This is where most of the work is. Instructors often switch the Files tab off too (/files answers 403), and slides then live only in modules, attachments and links. So file ids are gathered from every source and unioned:

1. the Files API, when it answers
2. module items of type File
3. attachments on assignments, discussions, announcements, and on individual replies
4. links in every HTML body: the syllabus, assignment descriptions, pages, discussion and announcement messages, every reply, and each module item's external URL

The link pass (extractFileRefs) walks a, img, source, iframe, embed and object elements for /courses/<c>/files/<id> and /files/<id>, then runs a regex for plain-text URLs the DOM walk missed. It keeps the course the link points into and the link's text, not just the id. That matters because instructors paste pages from older instances of a course, so a link often points at a file in another course.

Each id without metadata is then looked up in the course its link points into, then in this course, then globally (/files/:id). If none answers, the student has no access: the file lives in a course they aren't in, and nothing, not even the student clicking the link, can download it. It is kept as restricted, with its name and a link to open it in Canvas, rather than silently dropped.

Downloading tries, in order: the signed CDN link from the metadata, then …/files/:id/download?download_frd=1 under the linked course, this course, and no course. Each file is fingerprinted with SHA-256 (crypto.subtle), and its path in the zip mirrors Canvas's own folders.

Files hosted off Canvas. Pages often link straight to publisher or library PDFs. Links that aren't on Canvas and end in a document extension are collected, and fetched outside the page (the extension's service worker, or the app) with no cookies, so the student's session never reaches a third-party site. Each is capped at 40 MB; bigger ones stay a link.

6. Names stay out

By default the export writes a discussion reply's author as The lecturer (anyone on the course's teacher or TA lists) or Student A, B…, lettered per thread so a conversation still reads. course.json says how many teach the course, not who. Every name the export saw goes in _names.json, and Simeon's engine replaces those names in every text before any AI reads it, then deletes the file. A student can turn this off in the app's settings.

7. What comes out

HTML bodies become Markdown through a small DOM walker (headings, lists, tables, code, links, images; scripts and styles dropped), and each note starts with YAML frontmatter (type, course, url, dates) for the engine to read.

Simeon - <course>/
  Course Index.md          every module, assignment, page and topic, linked
  Map.md                   the modules as a Mermaid diagram
  Syllabus.md · Calendar.md · Grading.md
  Modules/ · Assignments/ · Pages/ · Discussions/ · Announcements/
  Files/                   every downloaded file, in Canvas's folders, with Files Index.md
  Files/External/          the files hosted off Canvas
  course.json              the course as data

course.json saves a build from working the course out of file names. It lists each module in order and, for each file, its module and position (from a module item, or from the page or assignment that linked it, when that is in a module), dates, size, fingerprint, and a guessed kind: lecture, seminar, exam, solutions, reading, case, assignment, course info or other, matched on the name, then the folder, then the module, in English, Swedish and Dutch. It marks duplicates (the same bytes twice), older versions (the same name without "v2", "final", "updated"…) and pairs each exam with its solutions by name. All of those are guesses, and the file says so; the agent that builds the plan is told to check a file before trusting what it's called.

The zip is built with JSZip as a blob and handed over with an object URL. Not base64 through chrome.downloads: extension messages cap at about 64 MB, and a course with its slides can be far bigger.

8. When something goes missing

Every export logs structured objects in the Canvas tab's console: what it found per source (Simeon file discovery: {…}, with files per source, links per body type, cross-course links, restricted and external), and download summaries. When files are missing, that object usually says why. All zeros means the material is in an external tool (a video platform, SCORM) that the Canvas API doesn't expose. On Windows each step is also written, with its time, to a log beside the app's settings, which shows which step was slow.

The whole chain is tested end to end: the real extension loaded into headless Chrome over the DevTools pipe, against a fake Canvas served over local HTTPS, with a Canvas tab opened before the extension, no Canvas tab at all, and a signed-out Canvas.

Reusing it: a recipe

For any tool that needs a student's (or employee's) data from a web app they're already signed in to:

1. Run inside the signed-in origin and call the app's own API with the session cookie. No tokens to issue, no admin to ask, and access exactly equal to what the user can already see.
2. Put every request through one polite function: a global concurrency cap, exponential back-off on 429 (and on whatever the app returns instead), and a pause when the rate-limit header runs low. Read-only.
3. Don't trust the obvious endpoint. Union every place an item can be referenced from, keep where each reference points, and resolve it from there first. List what you can't reach instead of dropping it.
4. Fetch third-party links outside the page, with no cookies.
5. Write one export and adapt the hosts. A small shim for messaging and file saving let the same script run in an extension, WebKit and WebView2 without a fork.
6. Answer at once, report as events. Long jobs outlive a single message.
7. Ship data alongside documents (course.json): what you learn while exporting is cheap to write down and expensive to guess later. Mark guesses as guesses.

Where to look in the code