MightyPork
/
a
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

new docs

Browse Source
repl
Ondřej Hruška 3 weeks ago
parent b61bf7b155
commit 4e7b25d825
2 changed files with 447 additions and 98 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 197
      docs/concurrency.md
  2. 348
      docs/http.md

197
docs/concurrency.md
Unescape View File

@ -1,20 +1,25 @@
# Concurrency: coroutines and `task` # Concurrency
`a` runs every script on an async core ([tokio](https://tokio.rs/)). The async `a` runs every script on an async core ([tokio](https://tokio.rs/)). The async
stdlib calls — `os.sleep`, the `http` client, `sqlite` queries — don't block the stdlib calls — `os.sleep`, the `http` client, `sqlite` queries — don't block the
OS thread while they wait; they suspend and let other work run. This page OS thread while they wait: they suspend and let other work run. This page covers
explains how that interacts with Lua coroutines, and how to run several pieces of how that interacts with Lua coroutines and how to run several pieces of work at
work concurrently with `task.join`. once with `task.join`.
## TL;DR Plain Lua coroutines behave exactly as in stock Lua. The moment a coroutine needs
to wait on something async — a sleep, a request, a query — run it through
`task.join` rather than driving it yourself, for the reason in
[Self-driven coroutines](#self-driven-coroutines).
- Plain Lua coroutines work exactly as in stock Lua. ```lua
- Async stdlib calls (`os.sleep`, `http.*`, `sqlite` …) suspend on the runtime — -- Three requests that would take ~3s back-to-back finish in ~1s.
they don't block the thread. local a, b, c = task.join(
- Use **`task.join`** to run multiple coroutines concurrently. N tasks that each function() return http.get("https://httpbingo.org/delay/1").status end,
wait T seconds finish in ~T, not ~N·T. function() return http.get("https://httpbingo.org/delay/1").status end,
- **Don't** call async stdlib functions inside a coroutine you drive yourself function() return http.get("https://httpbingo.org/delay/1").status end
with `coroutine.resume` / `coroutine.wrap` — see [the gotcha](#the-gotcha). )
print(a, b, c) --> 200 200 200
```
## Plain coroutines ## Plain coroutines
@ -22,22 +27,22 @@ Stock Lua coroutines are pure VM machinery and behave normally:
```lua ```lua
local function squares(n) local function squares(n)
for i = 1, n do for i = 1, n do
coroutine.yield(i * i) coroutine.yield(i * i)
end end
end end
local next = coroutine.wrap(squares) local gen = coroutine.wrap(squares)
print(next(3), next(3), next(3)) --> 1 4 9 print(gen(3), gen(), gen()) --> 1 4 9
``` ```
This is ordinary cooperative scheduling: nothing runs concurrently, and the This is ordinary cooperative scheduling: nothing runs concurrently, and the tokio
tokio runtime is never involved. runtime is never involved.
## Async calls suspend, they don't block ## Async calls suspend, they don't block
An async stdlib call at the top level of your script (or inside a `task` An async stdlib call — at the top level of your script, or inside a `task.join`
coroutine) suspends until it's ready, without tying up the thread: coroutine — suspends until it is ready without tying up the thread:
```lua ```lua
local t0 = os.microtime() local t0 = os.microtime()
@ -45,130 +50,126 @@ os.sleep(0.10)
print(string.format("waited %.3fs", os.microtime() - t0)) --> waited ~0.100s print(string.format("waited %.3fs", os.microtime() - t0)) --> waited ~0.100s
``` ```
While that sleep is pending, any sibling tasks (see below) keep making progress. While that sleep is pending, any sibling tasks keep making progress.
## `task.join` — run work concurrently ## Running work concurrently
``` ### `task.join(fn1, fn2, ...)`
task.join(fn1, fn2, ...) -> r1, r2, ...
```
Runs each function as its own coroutine, drives them **concurrently** on the Run each function as its own coroutine, drive them **concurrently** on the
runtime, and returns each one's first result positionally once all have runtime, and return each one's first result positionally once all have finished.
finished. If any task raises an error, `task.join` re-raises the first one. If a task raises an error, `task.join` re-raises the first one.
Because async calls suspend instead of blocking, the tasks overlap: Because async calls suspend instead of blocking, the tasks overlap — the
wall-clock time is the *longest* task, not the sum:
```lua ```lua
local function worker(name, secs) local function worker(name, secs)
return function() return function()
os.sleep(secs) -- suspends; siblings run meanwhile os.sleep(secs) -- suspends; siblings run meanwhile
return name return name
end end
end end
local t0 = os.microtime() local t0 = os.microtime()
local a, b, c = task.join( local a, b, c = task.join(
worker("slow", 0.30), worker("slow", 0.30),
worker("med", 0.20), worker("med", 0.20),
worker("fast", 0.10) worker("fast", 0.10)
) )
print(a, b, c) --> slow med fast print(a, b, c) --> slow med fast
print(string.format("%.3fs", os.microtime() - t0)) --> ~0.300s, not 0.600s print(string.format("%.3fs", os.microtime() - t0)) --> ~0.300s, not 0.600s
``` ```
The three sleeps run at the same time, so the wall-clock time is the *longest* ### Returning results
single task, not the sum.
### Real-world example: concurrent HTTP fetches
```lua
local function fetch(url)
return function()
return http.get(url).status
end
end
local s1, s2, s3 = task.join(
fetch("https://httpbingo.org/delay/1"),
fetch("https://httpbingo.org/delay/1"),
fetch("https://httpbingo.org/delay/1")
)
print(s1, s2, s3) --> 200 200 200, in ~1s total instead of ~3s
```
### Passing results back
Each task returns its first value to the corresponding slot: Each task's first return value lands in the matching slot, so several results come
back together:
```lua ```lua
local me, repos = task.join( local me, repos = task.join(
function() return http.getJSON("https://api.github.com/users/torvalds") end, function() return http.getJSON("https://api.github.com/users/torvalds") end,
function() return http.getJSON("https://api.github.com/users/torvalds/repos") end function() return http.getJSON("https://api.github.com/users/torvalds/repos") end
) )
print(me.name, #repos) -- both fetched concurrently print(me.name, #repos) -- both fetched concurrently
``` ```
### Nesting ### Nesting
`task.join` suspends like any other async call, so a task can itself call `task.join` suspends like any other async call, so a task may itself call
`task.join`: `task.join`:
```lua ```lua
local total = task.join( local total = task.join(
function() function()
local x, y = task.join( local x, y = task.join(
function() os.sleep(0.05); return 21 end, function() os.sleep(0.05); return 21 end,
function() os.sleep(0.05); return 21 end function() os.sleep(0.05); return 21 end
) )
return x + y -- 42 return x + y -- 42
end, end,
function() os.sleep(0.10); return "sibling" end function() os.sleep(0.10); return "sibling" end
) )
print(total) --> 42 print(total) --> 42
``` ```
## The gotcha ## Self-driven coroutines
Async stdlib functions only suspend correctly when the runtime is driving the Async stdlib functions suspend correctly only when the runtime is driving the
coroutine — i.e. at the top level of your script, or inside a `task.join` coroutine — at the top level of your script, or inside a `task.join` coroutine. If
coroutine. If you drive a coroutine **yourself**, the async call does *not* wait: you drive a coroutine **yourself** with `coroutine.resume` or `coroutine.wrap`, an
async call does *not* wait:
```lua ```lua
-- DON'T do this: -- DON'T do this:
local co = coroutine.wrap(function() local co = coroutine.wrap(function()
os.sleep(0.50) os.sleep(0.50)
return "awoke" return "awoke"
end) end)
local v = co() -- returns immediately with an opaque value; the 0.5s wait local v = co() -- returns immediately with an opaque value; the 0.5s wait never
-- never happens, because `coroutine.wrap` can't drive the -- happens, because coroutine.wrap can't drive the runtime.
-- runtime.
``` ```
Under the hood, an async call yields a private marker that only the runtime's Under the hood an async call yields a private marker that only the runtime's
scheduler understands; a hand-written `resume`/`wrap` loop just receives that scheduler understands; a hand-written `resume`/`wrap` loop just receives that
marker and moves on. The rule of thumb: marker and moves on.
**Rule of thumb:** use plain `coroutine.*` for pure-Lua generators; the moment a
coroutine needs to `os.sleep`, hit the network, or touch the database, run it
through `task.join` instead.
> Use plain `coroutine.*` for pure-Lua generators. The moment a coroutine needs ## How it works
> to `os.sleep`, hit the network, or touch the database, run it through
> `task.join` instead.
## How it works (and its limits) `task.join` wraps each function in a Lua coroutine and polls them all on the tokio
runtime. The concurrency comes from the **reactor** — timers and I/O yielding
control while they wait — not from extra threads.
`task.join` wraps each function in a Lua coroutine and polls them all - **Concurrency, not parallelism.** Everything runs on one thread; there are
concurrently on the tokio runtime. The concurrency comes from the **reactor** — simply many operations in flight at once.
timers and I/O yielding control while they wait — not from extra threads: - **I/O-bound work overlaps.** Sleeps, HTTP requests, and database queries all
wait at the same time.
- **CPU-bound work does not yield.** A tight compute loop with no async calls
blocks its siblings until it finishes or reaches an async call.
- Great for **I/O-bound** work: sleeps, HTTP requests, database queries all ## Full example
overlap.
- A **CPU-bound** task (a tight compute loop with no async calls) will *not*
yield, so it blocks its siblings until it finishes or hits an async call.
In other words this is **concurrency, not parallelism**: one thread, many ```lua
in-flight operations. -- Fetch several resources at once, then combine them.
local function get(url)
return function() return http.getJSON(url) end
end
## See also local user, repos = task.join(
get("https://api.github.com/users/torvalds"),
get("https://api.github.com/users/torvalds/repos")
)
print(user.name)
print(#repos .. " public repos")
-- Both requests ran concurrently: ~1 round-trip of latency, not 2.
```
- Runnable demo: [`lua/coroutines-demo.lua`](../lua/coroutines-demo.lua) A runnable version of these patterns ships in
(`a lua/coroutines-demo.lua`) [`lua/coroutines-demo.lua`](../lua/coroutines-demo.lua) — run it with
`a lua/coroutines-demo.lua`.

348
docs/http.md
Unescape View File

@ -0,0 +1,348 @@
# `http`
The `http` module is an HTTP/HTTPS client for calling APIs, scraping pages, and
submitting forms. TLS is compiled into the binary (rustls with the `ring`
provider), so HTTPS works with no system library. There are two ways in: plain
functions for one-off requests, and a **session** object that carries a cookie
jar across requests for stateful flows like logging in.
```lua
local resp = http.get("https://example.com")
print(resp.status, resp.ok) --> 200 true
print(resp.body) --> "<!doctype html>…"
-- JSON in one step
local data = http.getJSON("https://api.example.com/users")
for _, user in ipairs(data) do
print(user.name)
end
```
## Making requests
Every helper takes a URL and an optional `opts` table (see [Request
options](#request-options)) and returns a [response](#the-response) table.
```lua
http.get(url [, opts])
http.post(url [, opts])
http.put(url [, opts])
http.patch(url [, opts])
http.delete(url [, opts])
http.head(url [, opts])
```
These are thin shorthands over the one primitive:
### `http.request(method, url [, opts])`
Send a request with an explicit method (any verb, e.g. `"GET"`, `"OPTIONS"`) and
return the response. The shorthands above are just `http.request` with the method
filled in.
```lua
local resp = http.request("DELETE", "https://api.example.com/items/42")
```
## The response
Every request returns a table describing the response:
| Field | Type | Description |
|-----------|-----------|-------------------------------------------------------------------|
| `status` | integer | HTTP status code, e.g. `200`, `404`. |
| `ok` | boolean | `true` when `status` is in the 2xx range. |
| `headers` | table | Response headers, keyed by **lowercase** name. |
| `body` | string | The raw response body (Lua strings are byte sequences). |
| `json` | function | `resp.json()` parses `body` as JSON. See [JSON](#json). |
Header names are lowercased so you can look them up without guessing the server's
capitalization. If a header appears more than once, the first value wins.
```lua
local resp = http.get("https://example.com")
print(resp.headers["content-type"]) --> "text/html; charset=utf-8"
```
A non-2xx status is **not** an error — `resp.status` and `resp.ok` simply report
it. Only a failure to get a response at all (DNS, connection, TLS, timeout) raises
a Lua error. See [Errors](#errors).
## Request options
The optional `opts` table accepts these fields, all optional:
| Field | Type | Behaviour |
|-----------|--------|--------------------------------------------------------------------------|
| `headers` | table | Extra request headers, `{["X-Foo"] = "bar"}`. |
| `body` | string | Raw request body; set `Content-Type` yourself via `headers`. |
| `json` | any | Serialized to JSON; sets `Content-Type: application/json`. |
| `form` | table | URL-encoded; sets `Content-Type: application/x-www-form-urlencoded`. |
| `cookies` | table | Cookies for this request, `{session = "abc"}` → `Cookie:` header. |
| `timeout` | number | Per-request timeout in seconds (default `30`). |
| `auth` | table | Basic or digest credentials. See [Authentication](#authentication). |
### Bodies
`json`, `form`, and `body` are three ways to set the request body; if more than
one is given, the first present in that order wins.
```lua
-- JSON body (table serialized to an object)
http.post("https://api.example.com/items", { json = { name = "test", count = 5 } })
-- Form submission (application/x-www-form-urlencoded)
http.post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
-- Raw body with an explicit content type
http.post("https://example.com/ingest", {
body = "id,name\n1,alice\n",
headers = { ["Content-Type"] = "text/csv" },
})
```
`json` serializes the same way as `utils.toJSON`: a table with
sequential integer keys becomes a JSON array, otherwise an object, and
`utils.NULL` becomes JSON `null` (a literal Lua `nil` cannot live in
a table).
### Headers and cookies
```lua
local resp = http.get("https://example.com", {
headers = { ["Accept"] = "application/json", ["X-Token"] = "xyz" },
cookies = { session = "abc123" },
timeout = 10,
})
```
Per-request `cookies` are sent as a `Cookie` header. With a [session](#sessions),
they are merged with the jar and take precedence on a name collision.
## JSON
### `resp.json()`
Parse the response body as JSON and return the resulting Lua value. It is the
counterpart of `utils.fromJSON`: JSON objects become tables, arrays
become array-tables, and `null` becomes `utils.NULL`. Calling it on a
body that is not valid JSON raises an error.
```lua
local resp = http.post("https://api.example.com/echo", { json = { hello = "world" } })
print(resp.json().hello) --> "world"
```
### `http.getJSON(url [, opts])`
Shorthand for a GET that parses the body. Returns **two** values: the parsed body
and the full response.
```lua
local data, resp = http.getJSON("https://api.example.com/users")
print(resp.status, #data)
```
### `http.postJSON(url, body [, opts])`
Shorthand for a POST with a JSON body — equivalent to setting `opts.json = body`.
Returns the response.
```lua
local resp = http.postJSON("https://api.example.com/items", { name = "test", count = 5 })
if resp.ok then print(resp.json().id) end
```
## Authentication
Pass credentials in `opts.auth`. The `scheme` is `"basic"` (the default) or
`"digest"`.
| Field | Type | Description |
|------------|--------|----------------------------------------------|
| `username` | string | Required. |
| `password` | string | Required. |
| `scheme` | string | `"basic"` (default) or `"digest"`. |
```lua
-- HTTP Basic
local resp = http.get("https://api.example.com/private", {
auth = { username = "alice", password = "secret" },
})
-- HTTP Digest — the 401 challenge is answered automatically
local resp = http.get("https://api.example.com/private", {
auth = { username = "alice", password = "secret", scheme = "digest" },
})
```
For basic auth the `Authorization` header is sent with the request. For digest the
client sends the request, reads the server's `401` challenge, computes the
response, and retries once; you only see the final response. Auth works the same
way on [sessions](#sessions).
## Sessions
A **session** wraps its own cookie jar. Cookies from `Set-Cookie` responses are
stored automatically and sent back on later requests to matching hosts — which is
what makes login-then-fetch flows work. A session also has the same request
methods as the plain module.
### `http.session([path])`
Create a session. With a `path`, the cookie jar is preloaded from that file (see
[Persisting the jar](#persisting-the-jar)).
```lua
local s = http.session() -- fresh, empty jar
local s = http.session("cookies.jsonl") -- jar loaded from disk
```
### Requests
A session has every method the plain module has — `request`, `get`, `post`,
`put`, `patch`, `delete`, `head`, `getJSON`, `postJSON` — called with `:` syntax
and accepting the same `opts`:
```lua
local s = http.session()
-- Log in; the response's Set-Cookie is captured into the jar
s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
-- The session cookie is sent automatically
local page = s:get("https://example.com/dashboard")
```
### Cookie behaviour
A stored cookie is sent on a request when the request host **equals** the cookie's
domain or is a subdomain of it. The jar reads the `name=value` pair and the
`Domain` attribute from each `Set-Cookie` header; other attributes (`Path`,
`Expires`, `Secure`, `HttpOnly`) are ignored. When no `Domain` is given, the
request host is used.
`Set-Cookie` headers set on redirect responses are captured too, so a login that
`302`-redirects to a dashboard still records its cookie.
### Inspecting and clearing
#### `s:cookies()`
Return the jar as a nested table, `{domain = {name = value}}`, for inspection.
```lua
local jar = s:cookies()
for domain, names in pairs(jar) do
for name, value in pairs(names) do
print(domain, name, value)
end
end
```
#### `s:clearCookies()`
Empty the in-memory jar.
### Persisting the jar
Cookies live in memory for the session's lifetime. Save them to reuse a logged-in
session across script runs.
#### `s:save(path)`
Write the jar to `path` as JSONL — one JSON object per line, one cookie per line:
```jsonl
{"domain":"example.com","name":"session","value":"abc123"}
{"domain":"api.example.com","name":"token","value":"xyz789"}
```
#### `s:load(path)`
Merge cookies from a JSONL file into the current jar (existing cookies are kept,
matching names overwritten). `http.session(path)` is the same as creating a
session and calling `:load(path)`.
```lua
-- First run: log in and persist
local s = http.session()
s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
s:save("session.jsonl")
-- Later run: restore and continue without logging in again
local s = http.session("session.jsonl")
local page = s:get("https://example.com/dashboard")
```
## Redirects
Redirects are followed automatically, up to 10 hops; you receive the final
response. A `303`, and a `301`/`302` in response to a `POST`, are followed as a
bodyless `GET`, matching browser behaviour. For a session, cookies set along the
way are captured at each hop.
## Errors
Failing to obtain a response raises a Lua error: DNS failure, connection refused,
a TLS problem, or a timeout. An HTTP error *status* (4xx/5xx) does not — it is
reported through `resp.status`/`resp.ok`. Parsing a non-JSON body with
`resp.json()` also raises.
Wrap calls in `pcall` or `utils.try` where you want to handle failure
rather than abort:
```lua
local resp, err = utils.try(function()
return http.get("https://does-not-exist.invalid", { timeout = 5 })
end)
if not resp then
log.error("request failed: " .. tostring(err))
end
```
## Notes
- **HTTPS needs no setup.** The TLS stack (rustls + `ring`) is compiled in; trust
roots come from the system certificate store.
- **A default `User-Agent` is sent** (`a/<version>`) because some servers reject
requests without one. Override it with a `User-Agent` entry in `opts.headers`.
- **Requests don't block the event loop.** Network I/O runs on the async core, so
a slow request does not stall other async work (timers, `os.sleep`, SQLite) in
the same script.
- **The cookie jar is per session.** Plain `http.get`/`http.post` calls do not
retain cookies between calls; use a session for that.
## Full example
```lua
-- Talk to a JSON API with a bearer token, then drive a stateful session.
-- One-off authenticated JSON call
local items, resp = http.getJSON("https://api.example.com/items", {
headers = { ["Authorization"] = "Bearer " .. token },
timeout = 15,
})
if not resp.ok then
error("list failed: HTTP " .. resp.status)
end
for _, item in ipairs(items) do
print(item.id, item.name)
end
-- Create one
local created = http.postJSON("https://api.example.com/items", { name = "widget" }, {
headers = { ["Authorization"] = "Bearer " .. token },
})
print("created id:", created.json().id)
-- A login session that persists across runs
local s = http.session("session.jsonl") -- restore if present
local home = s:get("https://example.com/dashboard")
if home.status == 401 then -- session expired; log in again
s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
home = s:get("https://example.com/dashboard")
s:save("session.jsonl")
end
print(home.ok and "logged in" or "login failed")
```
Write Preview
Loading…
Cancel
Save