diff --git a/docs/concurrency.md b/docs/concurrency.md index 4ab86a7..1ed7c04 100644 --- a/docs/concurrency.md +++ b/docs/concurrency.md @@ -1,20 +1,25 @@ -# Concurrency: coroutines and `task` +# Concurrency `a` runs every script on an async core ([tokio](https://tokio.rs/)). The async stdlib calls — `os.sleep`, the `http` client, `sqlite` queries — don't block the -OS thread while they wait; they suspend and let other work run. This page -explains how that interacts with Lua coroutines, and how to run several pieces of -work concurrently with `task.join`. +OS thread while they wait: they suspend and let other work run. This page covers +how that interacts with Lua coroutines and how to run several pieces of work at +once with `task.join`. -## TL;DR +Plain Lua coroutines behave exactly as in stock Lua. The moment a coroutine needs +to wait on something async — a sleep, a request, a query — run it through +`task.join` rather than driving it yourself, for the reason in +[Self-driven coroutines](#self-driven-coroutines). -- Plain Lua coroutines work exactly as in stock Lua. -- Async stdlib calls (`os.sleep`, `http.*`, `sqlite` …) suspend on the runtime — - they don't block the thread. -- Use **`task.join`** to run multiple coroutines concurrently. N tasks that each - wait T seconds finish in ~T, not ~N·T. -- **Don't** call async stdlib functions inside a coroutine you drive yourself - with `coroutine.resume` / `coroutine.wrap` — see [the gotcha](#the-gotcha). +```lua +-- Three requests that would take ~3s back-to-back finish in ~1s. +local a, b, c = task.join( + function() return http.get("https://httpbingo.org/delay/1").status end, + function() return http.get("https://httpbingo.org/delay/1").status end, + function() return http.get("https://httpbingo.org/delay/1").status end +) +print(a, b, c) --> 200 200 200 +``` ## Plain coroutines @@ -22,22 +27,22 @@ Stock Lua coroutines are pure VM machinery and behave normally: ```lua local function squares(n) - for i = 1, n do - coroutine.yield(i * i) - end + for i = 1, n do + coroutine.yield(i * i) + end end -local next = coroutine.wrap(squares) -print(next(3), next(3), next(3)) --> 1 4 9 +local gen = coroutine.wrap(squares) +print(gen(3), gen(), gen()) --> 1 4 9 ``` -This is ordinary cooperative scheduling: nothing runs concurrently, and the -tokio runtime is never involved. +This is ordinary cooperative scheduling: nothing runs concurrently, and the tokio +runtime is never involved. ## Async calls suspend, they don't block -An async stdlib call at the top level of your script (or inside a `task` -coroutine) suspends until it's ready, without tying up the thread: +An async stdlib call — at the top level of your script, or inside a `task.join` +coroutine — suspends until it is ready without tying up the thread: ```lua local t0 = os.microtime() @@ -45,130 +50,126 @@ os.sleep(0.10) print(string.format("waited %.3fs", os.microtime() - t0)) --> waited ~0.100s ``` -While that sleep is pending, any sibling tasks (see below) keep making progress. +While that sleep is pending, any sibling tasks keep making progress. -## `task.join` — run work concurrently +## Running work concurrently -``` -task.join(fn1, fn2, ...) -> r1, r2, ... -``` +### `task.join(fn1, fn2, ...)` -Runs each function as its own coroutine, drives them **concurrently** on the -runtime, and returns each one's first result positionally once all have -finished. If any task raises an error, `task.join` re-raises the first one. +Run each function as its own coroutine, drive them **concurrently** on the +runtime, and return each one's first result positionally once all have finished. +If a task raises an error, `task.join` re-raises the first one. -Because async calls suspend instead of blocking, the tasks overlap: +Because async calls suspend instead of blocking, the tasks overlap — the +wall-clock time is the *longest* task, not the sum: ```lua local function worker(name, secs) - return function() - os.sleep(secs) -- suspends; siblings run meanwhile - return name - end + return function() + os.sleep(secs) -- suspends; siblings run meanwhile + return name + end end local t0 = os.microtime() local a, b, c = task.join( - worker("slow", 0.30), - worker("med", 0.20), - worker("fast", 0.10) + worker("slow", 0.30), + worker("med", 0.20), + worker("fast", 0.10) ) -print(a, b, c) --> slow med fast +print(a, b, c) --> slow med fast print(string.format("%.3fs", os.microtime() - t0)) --> ~0.300s, not 0.600s ``` -The three sleeps run at the same time, so the wall-clock time is the *longest* -single task, not the sum. - -### Real-world example: concurrent HTTP fetches - -```lua -local function fetch(url) - return function() - return http.get(url).status - end -end - -local s1, s2, s3 = task.join( - fetch("https://httpbingo.org/delay/1"), - fetch("https://httpbingo.org/delay/1"), - fetch("https://httpbingo.org/delay/1") -) -print(s1, s2, s3) --> 200 200 200, in ~1s total instead of ~3s -``` - -### Passing results back +### Returning results -Each task returns its first value to the corresponding slot: +Each task's first return value lands in the matching slot, so several results come +back together: ```lua local me, repos = task.join( - function() return http.getJSON("https://api.github.com/users/torvalds") end, - function() return http.getJSON("https://api.github.com/users/torvalds/repos") end + function() return http.getJSON("https://api.github.com/users/torvalds") end, + function() return http.getJSON("https://api.github.com/users/torvalds/repos") end ) print(me.name, #repos) -- both fetched concurrently ``` ### Nesting -`task.join` suspends like any other async call, so a task can itself call +`task.join` suspends like any other async call, so a task may itself call `task.join`: ```lua local total = task.join( - function() - local x, y = task.join( - function() os.sleep(0.05); return 21 end, - function() os.sleep(0.05); return 21 end - ) - return x + y -- 42 - end, - function() os.sleep(0.10); return "sibling" end + function() + local x, y = task.join( + function() os.sleep(0.05); return 21 end, + function() os.sleep(0.05); return 21 end + ) + return x + y -- 42 + end, + function() os.sleep(0.10); return "sibling" end ) print(total) --> 42 ``` -## The gotcha +## Self-driven coroutines -Async stdlib functions only suspend correctly when the runtime is driving the -coroutine — i.e. at the top level of your script, or inside a `task.join` -coroutine. If you drive a coroutine **yourself**, the async call does *not* wait: +Async stdlib functions suspend correctly only when the runtime is driving the +coroutine — at the top level of your script, or inside a `task.join` coroutine. If +you drive a coroutine **yourself** with `coroutine.resume` or `coroutine.wrap`, an +async call does *not* wait: ```lua -- DON'T do this: local co = coroutine.wrap(function() - os.sleep(0.50) - return "awoke" + os.sleep(0.50) + return "awoke" end) -local v = co() -- returns immediately with an opaque value; the 0.5s wait - -- never happens, because `coroutine.wrap` can't drive the - -- runtime. +local v = co() -- returns immediately with an opaque value; the 0.5s wait never + -- happens, because coroutine.wrap can't drive the runtime. ``` -Under the hood, an async call yields a private marker that only the runtime's +Under the hood an async call yields a private marker that only the runtime's scheduler understands; a hand-written `resume`/`wrap` loop just receives that -marker and moves on. The rule of thumb: +marker and moves on. + +**Rule of thumb:** use plain `coroutine.*` for pure-Lua generators; the moment a +coroutine needs to `os.sleep`, hit the network, or touch the database, run it +through `task.join` instead. -> Use plain `coroutine.*` for pure-Lua generators. The moment a coroutine needs -> to `os.sleep`, hit the network, or touch the database, run it through -> `task.join` instead. +## How it works -## How it works (and its limits) +`task.join` wraps each function in a Lua coroutine and polls them all on the tokio +runtime. The concurrency comes from the **reactor** — timers and I/O yielding +control while they wait — not from extra threads. -`task.join` wraps each function in a Lua coroutine and polls them all -concurrently on the tokio runtime. The concurrency comes from the **reactor** — -timers and I/O yielding control while they wait — not from extra threads: +- **Concurrency, not parallelism.** Everything runs on one thread; there are + simply many operations in flight at once. +- **I/O-bound work overlaps.** Sleeps, HTTP requests, and database queries all + wait at the same time. +- **CPU-bound work does not yield.** A tight compute loop with no async calls + blocks its siblings until it finishes or reaches an async call. -- Great for **I/O-bound** work: sleeps, HTTP requests, database queries all - overlap. -- A **CPU-bound** task (a tight compute loop with no async calls) will *not* - yield, so it blocks its siblings until it finishes or hits an async call. +## Full example -In other words this is **concurrency, not parallelism**: one thread, many -in-flight operations. +```lua +-- Fetch several resources at once, then combine them. +local function get(url) + return function() return http.getJSON(url) end +end -## See also +local user, repos = task.join( + get("https://api.github.com/users/torvalds"), + get("https://api.github.com/users/torvalds/repos") +) + +print(user.name) +print(#repos .. " public repos") +-- Both requests ran concurrently: ~1 round-trip of latency, not 2. +``` -- Runnable demo: [`lua/coroutines-demo.lua`](../lua/coroutines-demo.lua) - (`a lua/coroutines-demo.lua`) +A runnable version of these patterns ships in +[`lua/coroutines-demo.lua`](../lua/coroutines-demo.lua) — run it with +`a lua/coroutines-demo.lua`. diff --git a/docs/http.md b/docs/http.md new file mode 100644 index 0000000..47b0b09 --- /dev/null +++ b/docs/http.md @@ -0,0 +1,348 @@ +# `http` + +The `http` module is an HTTP/HTTPS client for calling APIs, scraping pages, and +submitting forms. TLS is compiled into the binary (rustls with the `ring` +provider), so HTTPS works with no system library. There are two ways in: plain +functions for one-off requests, and a **session** object that carries a cookie +jar across requests for stateful flows like logging in. + +```lua +local resp = http.get("https://example.com") +print(resp.status, resp.ok) --> 200 true +print(resp.body) --> "…" + +-- JSON in one step +local data = http.getJSON("https://api.example.com/users") +for _, user in ipairs(data) do + print(user.name) +end +``` + +## Making requests + +Every helper takes a URL and an optional `opts` table (see [Request +options](#request-options)) and returns a [response](#the-response) table. + +```lua +http.get(url [, opts]) +http.post(url [, opts]) +http.put(url [, opts]) +http.patch(url [, opts]) +http.delete(url [, opts]) +http.head(url [, opts]) +``` + +These are thin shorthands over the one primitive: + +### `http.request(method, url [, opts])` + +Send a request with an explicit method (any verb, e.g. `"GET"`, `"OPTIONS"`) and +return the response. The shorthands above are just `http.request` with the method +filled in. + +```lua +local resp = http.request("DELETE", "https://api.example.com/items/42") +``` + +## The response + +Every request returns a table describing the response: + +| Field | Type | Description | +|-----------|-----------|-------------------------------------------------------------------| +| `status` | integer | HTTP status code, e.g. `200`, `404`. | +| `ok` | boolean | `true` when `status` is in the 2xx range. | +| `headers` | table | Response headers, keyed by **lowercase** name. | +| `body` | string | The raw response body (Lua strings are byte sequences). | +| `json` | function | `resp.json()` parses `body` as JSON. See [JSON](#json). | + +Header names are lowercased so you can look them up without guessing the server's +capitalization. If a header appears more than once, the first value wins. + +```lua +local resp = http.get("https://example.com") +print(resp.headers["content-type"]) --> "text/html; charset=utf-8" +``` + +A non-2xx status is **not** an error — `resp.status` and `resp.ok` simply report +it. Only a failure to get a response at all (DNS, connection, TLS, timeout) raises +a Lua error. See [Errors](#errors). + +## Request options + +The optional `opts` table accepts these fields, all optional: + +| Field | Type | Behaviour | +|-----------|--------|--------------------------------------------------------------------------| +| `headers` | table | Extra request headers, `{["X-Foo"] = "bar"}`. | +| `body` | string | Raw request body; set `Content-Type` yourself via `headers`. | +| `json` | any | Serialized to JSON; sets `Content-Type: application/json`. | +| `form` | table | URL-encoded; sets `Content-Type: application/x-www-form-urlencoded`. | +| `cookies` | table | Cookies for this request, `{session = "abc"}` → `Cookie:` header. | +| `timeout` | number | Per-request timeout in seconds (default `30`). | +| `auth` | table | Basic or digest credentials. See [Authentication](#authentication). | + +### Bodies + +`json`, `form`, and `body` are three ways to set the request body; if more than +one is given, the first present in that order wins. + +```lua +-- JSON body (table serialized to an object) +http.post("https://api.example.com/items", { json = { name = "test", count = 5 } }) + +-- Form submission (application/x-www-form-urlencoded) +http.post("https://example.com/login", { form = { user = "alice", pass = "secret" } }) + +-- Raw body with an explicit content type +http.post("https://example.com/ingest", { + body = "id,name\n1,alice\n", + headers = { ["Content-Type"] = "text/csv" }, +}) +``` + +`json` serializes the same way as `utils.toJSON`: a table with +sequential integer keys becomes a JSON array, otherwise an object, and +`utils.NULL` becomes JSON `null` (a literal Lua `nil` cannot live in +a table). + +### Headers and cookies + +```lua +local resp = http.get("https://example.com", { + headers = { ["Accept"] = "application/json", ["X-Token"] = "xyz" }, + cookies = { session = "abc123" }, + timeout = 10, +}) +``` + +Per-request `cookies` are sent as a `Cookie` header. With a [session](#sessions), +they are merged with the jar and take precedence on a name collision. + +## JSON + +### `resp.json()` + +Parse the response body as JSON and return the resulting Lua value. It is the +counterpart of `utils.fromJSON`: JSON objects become tables, arrays +become array-tables, and `null` becomes `utils.NULL`. Calling it on a +body that is not valid JSON raises an error. + +```lua +local resp = http.post("https://api.example.com/echo", { json = { hello = "world" } }) +print(resp.json().hello) --> "world" +``` + +### `http.getJSON(url [, opts])` + +Shorthand for a GET that parses the body. Returns **two** values: the parsed body +and the full response. + +```lua +local data, resp = http.getJSON("https://api.example.com/users") +print(resp.status, #data) +``` + +### `http.postJSON(url, body [, opts])` + +Shorthand for a POST with a JSON body — equivalent to setting `opts.json = body`. +Returns the response. + +```lua +local resp = http.postJSON("https://api.example.com/items", { name = "test", count = 5 }) +if resp.ok then print(resp.json().id) end +``` + +## Authentication + +Pass credentials in `opts.auth`. The `scheme` is `"basic"` (the default) or +`"digest"`. + +| Field | Type | Description | +|------------|--------|----------------------------------------------| +| `username` | string | Required. | +| `password` | string | Required. | +| `scheme` | string | `"basic"` (default) or `"digest"`. | + +```lua +-- HTTP Basic +local resp = http.get("https://api.example.com/private", { + auth = { username = "alice", password = "secret" }, +}) + +-- HTTP Digest — the 401 challenge is answered automatically +local resp = http.get("https://api.example.com/private", { + auth = { username = "alice", password = "secret", scheme = "digest" }, +}) +``` + +For basic auth the `Authorization` header is sent with the request. For digest the +client sends the request, reads the server's `401` challenge, computes the +response, and retries once; you only see the final response. Auth works the same +way on [sessions](#sessions). + +## Sessions + +A **session** wraps its own cookie jar. Cookies from `Set-Cookie` responses are +stored automatically and sent back on later requests to matching hosts — which is +what makes login-then-fetch flows work. A session also has the same request +methods as the plain module. + +### `http.session([path])` + +Create a session. With a `path`, the cookie jar is preloaded from that file (see +[Persisting the jar](#persisting-the-jar)). + +```lua +local s = http.session() -- fresh, empty jar +local s = http.session("cookies.jsonl") -- jar loaded from disk +``` + +### Requests + +A session has every method the plain module has — `request`, `get`, `post`, +`put`, `patch`, `delete`, `head`, `getJSON`, `postJSON` — called with `:` syntax +and accepting the same `opts`: + +```lua +local s = http.session() + +-- Log in; the response's Set-Cookie is captured into the jar +s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } }) + +-- The session cookie is sent automatically +local page = s:get("https://example.com/dashboard") +``` + +### Cookie behaviour + +A stored cookie is sent on a request when the request host **equals** the cookie's +domain or is a subdomain of it. The jar reads the `name=value` pair and the +`Domain` attribute from each `Set-Cookie` header; other attributes (`Path`, +`Expires`, `Secure`, `HttpOnly`) are ignored. When no `Domain` is given, the +request host is used. + +`Set-Cookie` headers set on redirect responses are captured too, so a login that +`302`-redirects to a dashboard still records its cookie. + +### Inspecting and clearing + +#### `s:cookies()` + +Return the jar as a nested table, `{domain = {name = value}}`, for inspection. + +```lua +local jar = s:cookies() +for domain, names in pairs(jar) do + for name, value in pairs(names) do + print(domain, name, value) + end +end +``` + +#### `s:clearCookies()` + +Empty the in-memory jar. + +### Persisting the jar + +Cookies live in memory for the session's lifetime. Save them to reuse a logged-in +session across script runs. + +#### `s:save(path)` + +Write the jar to `path` as JSONL — one JSON object per line, one cookie per line: + +```jsonl +{"domain":"example.com","name":"session","value":"abc123"} +{"domain":"api.example.com","name":"token","value":"xyz789"} +``` + +#### `s:load(path)` + +Merge cookies from a JSONL file into the current jar (existing cookies are kept, +matching names overwritten). `http.session(path)` is the same as creating a +session and calling `:load(path)`. + +```lua +-- First run: log in and persist +local s = http.session() +s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } }) +s:save("session.jsonl") + +-- Later run: restore and continue without logging in again +local s = http.session("session.jsonl") +local page = s:get("https://example.com/dashboard") +``` + +## Redirects + +Redirects are followed automatically, up to 10 hops; you receive the final +response. A `303`, and a `301`/`302` in response to a `POST`, are followed as a +bodyless `GET`, matching browser behaviour. For a session, cookies set along the +way are captured at each hop. + +## Errors + +Failing to obtain a response raises a Lua error: DNS failure, connection refused, +a TLS problem, or a timeout. An HTTP error *status* (4xx/5xx) does not — it is +reported through `resp.status`/`resp.ok`. Parsing a non-JSON body with +`resp.json()` also raises. + +Wrap calls in `pcall` or `utils.try` where you want to handle failure +rather than abort: + +```lua +local resp, err = utils.try(function() + return http.get("https://does-not-exist.invalid", { timeout = 5 }) +end) +if not resp then + log.error("request failed: " .. tostring(err)) +end +``` + +## Notes + +- **HTTPS needs no setup.** The TLS stack (rustls + `ring`) is compiled in; trust + roots come from the system certificate store. +- **A default `User-Agent` is sent** (`a/`) because some servers reject + requests without one. Override it with a `User-Agent` entry in `opts.headers`. +- **Requests don't block the event loop.** Network I/O runs on the async core, so + a slow request does not stall other async work (timers, `os.sleep`, SQLite) in + the same script. +- **The cookie jar is per session.** Plain `http.get`/`http.post` calls do not + retain cookies between calls; use a session for that. + +## Full example + +```lua +-- Talk to a JSON API with a bearer token, then drive a stateful session. + +-- One-off authenticated JSON call +local items, resp = http.getJSON("https://api.example.com/items", { + headers = { ["Authorization"] = "Bearer " .. token }, + timeout = 15, +}) +if not resp.ok then + error("list failed: HTTP " .. resp.status) +end +for _, item in ipairs(items) do + print(item.id, item.name) +end + +-- Create one +local created = http.postJSON("https://api.example.com/items", { name = "widget" }, { + headers = { ["Authorization"] = "Bearer " .. token }, +}) +print("created id:", created.json().id) + +-- A login session that persists across runs +local s = http.session("session.jsonl") -- restore if present +local home = s:get("https://example.com/dashboard") +if home.status == 401 then -- session expired; log in again + s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } }) + home = s:get("https://example.com/dashboard") + s:save("session.jsonl") +end +print(home.ok and "logged in" or "login failed") +```