You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
371 lines
13 KiB
371 lines
13 KiB
# `http`
|
|
|
|
The `http` module is an HTTP/HTTPS client for calling APIs, scraping pages, and
|
|
submitting forms. TLS is compiled into the binary (rustls with the `ring`
|
|
provider), so HTTPS works with no system library. There are two ways in: plain
|
|
functions for one-off requests, and a **session** object that carries a cookie
|
|
jar across requests for stateful flows like logging in.
|
|
|
|
```lua
|
|
local resp = http.get("https://example.com")
|
|
print(resp.status, resp.ok) --> 200 true
|
|
print(resp.body) --> "<!doctype html>…"
|
|
|
|
-- JSON in one step
|
|
local data = http.getJSON("https://api.example.com/users")
|
|
for _, user in ipairs(data) do
|
|
print(user.name)
|
|
end
|
|
```
|
|
|
|
## Making requests
|
|
|
|
Every helper takes a URL and an optional `opts` table (see [Request
|
|
options](#request-options)) and returns a [response](#the-response) table.
|
|
|
|
```lua
|
|
http.get(url [, opts])
|
|
http.post(url [, opts])
|
|
http.put(url [, opts])
|
|
http.patch(url [, opts])
|
|
http.delete(url [, opts])
|
|
http.head(url [, opts])
|
|
```
|
|
|
|
These are thin shorthands over the one primitive:
|
|
|
|
### `http.request(method, url [, opts])`
|
|
|
|
Send a request with an explicit method (any verb, e.g. `"GET"`, `"OPTIONS"`) and
|
|
return the response. The shorthands above are just `http.request` with the method
|
|
filled in.
|
|
|
|
```lua
|
|
local resp = http.request("DELETE", "https://api.example.com/items/42")
|
|
```
|
|
|
|
## The response
|
|
|
|
Every request returns a table describing the response:
|
|
|
|
| Field | Type | Description |
|
|
|--------------|-----------|-------------------------------------------------------------------|
|
|
| `status` | integer | HTTP status code, e.g. `200`, `404`. |
|
|
| `ok` | boolean | `true` when `status` is in the 2xx range. |
|
|
| `url` | string | The final URL, after any redirects. |
|
|
| `headers` | table | Response headers, keyed by **lowercase** name. |
|
|
| `setCookies` | table | Every `Set-Cookie` value on the final response, as an array. |
|
|
| `body` | string | The raw response body (Lua strings are byte sequences). |
|
|
| `json` | function | `resp.json()` parses `body` as JSON. See [JSON](#json). |
|
|
|
|
Header names are lowercased so you can look them up without guessing the server's
|
|
capitalization. A header that appears more than once has its values joined with
|
|
`", "` (the HTTP list convention) — except `set-cookie`, whose values can contain
|
|
commas: `headers["set-cookie"]` holds only the first one, and the full list is in
|
|
`resp.setCookies`.
|
|
|
|
```lua
|
|
local resp = http.get("https://example.com")
|
|
print(resp.headers["content-type"]) --> "text/html; charset=utf-8"
|
|
```
|
|
|
|
A non-2xx status is **not** an error — `resp.status` and `resp.ok` simply report
|
|
it. Only a failure to get a response at all (DNS, connection, TLS, timeout) raises
|
|
a Lua error. See [Errors](#errors).
|
|
|
|
## Request options
|
|
|
|
The optional `opts` table accepts these fields, all optional:
|
|
|
|
| Field | Type | Behaviour |
|
|
|-----------|--------|--------------------------------------------------------------------------|
|
|
| `headers` | table | Extra request headers, `{["X-Foo"] = "bar"}`. |
|
|
| `body` | string | Raw request body; set `Content-Type` yourself via `headers`. |
|
|
| `json` | any | Serialized to JSON; sets `Content-Type: application/json`. |
|
|
| `form` | table | URL-encoded; sets `Content-Type: application/x-www-form-urlencoded`. |
|
|
| `cookies` | table | Cookies for this request, `{session = "abc"}` → `Cookie:` header. |
|
|
| `timeout` | number | Timeout in seconds for the whole request, redirects included (default `30`). |
|
|
| `auth` | table | Basic or digest credentials. See [Authentication](#authentication). |
|
|
|
|
### Bodies
|
|
|
|
`json`, `form`, and `body` are three ways to set the request body; if more than
|
|
one is given, the first present in that order wins.
|
|
|
|
```lua
|
|
-- JSON body (table serialized to an object)
|
|
http.post("https://api.example.com/items", { json = { name = "test", count = 5 } })
|
|
|
|
-- Form submission (application/x-www-form-urlencoded)
|
|
http.post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
|
|
|
|
-- Raw body with an explicit content type
|
|
http.post("https://example.com/ingest", {
|
|
body = "id,name\n1,alice\n",
|
|
headers = { ["Content-Type"] = "text/csv" },
|
|
})
|
|
```
|
|
|
|
`json` serializes the same way as `utils.toJSON`: a table with
|
|
sequential integer keys becomes a JSON array, otherwise an object, and
|
|
`utils.NULL` becomes JSON `null` (a literal Lua `nil` cannot live in
|
|
a table).
|
|
|
|
### Headers and cookies
|
|
|
|
```lua
|
|
local resp = http.get("https://example.com", {
|
|
headers = { ["Accept"] = "application/json", ["X-Token"] = "xyz" },
|
|
cookies = { session = "abc123" },
|
|
timeout = 10,
|
|
})
|
|
```
|
|
|
|
Per-request `cookies` are sent as a `Cookie` header. With a [session](#sessions),
|
|
they are merged with the jar and take precedence on a name collision. Passing
|
|
`cookies` *and* a `Cookie` entry in `headers` is ambiguous and raises an error —
|
|
pick one.
|
|
|
|
## JSON
|
|
|
|
### `resp.json()`
|
|
|
|
Parse the response body as JSON and return the resulting Lua value. It is the
|
|
counterpart of `utils.fromJSON`: JSON objects become tables, arrays
|
|
become array-tables, and `null` becomes `utils.NULL`. Calling it on a
|
|
body that is not valid JSON raises an error.
|
|
|
|
```lua
|
|
local resp = http.post("https://api.example.com/echo", { json = { hello = "world" } })
|
|
print(resp.json().hello) --> "world"
|
|
```
|
|
|
|
### `http.getJSON(url [, opts])`
|
|
|
|
Shorthand for a GET that parses the body. Returns **two** values: the parsed body
|
|
and the full response.
|
|
|
|
```lua
|
|
local data, resp = http.getJSON("https://api.example.com/users")
|
|
print(resp.status, #data)
|
|
```
|
|
|
|
### `http.postJSON(url, body [, opts])`
|
|
|
|
Shorthand for a POST with a JSON body — equivalent to setting `opts.json = body`.
|
|
Returns the response.
|
|
|
|
```lua
|
|
local resp = http.postJSON("https://api.example.com/items", { name = "test", count = 5 })
|
|
if resp.ok then print(resp.json().id) end
|
|
```
|
|
|
|
## Authentication
|
|
|
|
Pass credentials in `opts.auth`. The `scheme` is `"basic"` (the default) or
|
|
`"digest"`.
|
|
|
|
| Field | Type | Description |
|
|
|------------|--------|----------------------------------------------|
|
|
| `username` | string | Required. |
|
|
| `password` | string | Required. |
|
|
| `scheme` | string | `"basic"` (default) or `"digest"`. |
|
|
|
|
```lua
|
|
-- HTTP Basic
|
|
local resp = http.get("https://api.example.com/private", {
|
|
auth = { username = "alice", password = "secret" },
|
|
})
|
|
|
|
-- HTTP Digest — the 401 challenge is answered automatically
|
|
local resp = http.get("https://api.example.com/private", {
|
|
auth = { username = "alice", password = "secret", scheme = "digest" },
|
|
})
|
|
```
|
|
|
|
For basic auth the `Authorization` header is sent with the request. For digest the
|
|
client sends the request, reads the server's `401` challenge, computes the
|
|
response, and retries once; you only see the final response. Auth works the same
|
|
way on [sessions](#sessions).
|
|
|
|
## Sessions
|
|
|
|
A **session** wraps its own cookie jar. Cookies from `Set-Cookie` responses are
|
|
stored automatically and sent back on later requests to matching hosts — which is
|
|
what makes login-then-fetch flows work. A session also has the same request
|
|
methods as the plain module.
|
|
|
|
### `http.session([path])`
|
|
|
|
Create a session. With a `path`, the cookie jar is preloaded from that file (see
|
|
[Persisting the jar](#persisting-the-jar)).
|
|
|
|
```lua
|
|
local s = http.session() -- fresh, empty jar
|
|
local s = http.session("cookies.jsonl") -- jar loaded from disk
|
|
```
|
|
|
|
### Requests
|
|
|
|
A session has every method the plain module has — `request`, `get`, `post`,
|
|
`put`, `patch`, `delete`, `head`, `getJSON`, `postJSON` — called with `:` syntax
|
|
and accepting the same `opts`:
|
|
|
|
```lua
|
|
local s = http.session()
|
|
|
|
-- Log in; the response's Set-Cookie is captured into the jar
|
|
s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
|
|
|
|
-- The session cookie is sent automatically
|
|
local page = s:get("https://example.com/dashboard")
|
|
```
|
|
|
|
### Cookie behaviour
|
|
|
|
The jar implements RFC 6265 the way a browser does (it is the `cookie_store`
|
|
crate underneath): `Domain`, `Path`, `Expires`, `Max-Age`, `Secure` and
|
|
`HttpOnly` attributes all take effect. In particular:
|
|
|
|
- A cookie is sent only to hosts the `Domain` attribute covers (the setting
|
|
host itself when absent), only on matching paths, and only until it expires —
|
|
a server deleting a cookie with `Max-Age=0` really removes it from the jar.
|
|
- A `Secure` cookie is never sent over plain `http://`.
|
|
- A response cannot set a cookie for an unrelated domain, nor for a public
|
|
suffix like `com` or `co.uk` — the jar checks Mozilla's Public Suffix List
|
|
(compiled into the binary), same as browsers and curl.
|
|
|
|
`Set-Cookie` headers on redirect responses are captured too, so a login that
|
|
`302`-redirects to a dashboard still records its cookie — and the redirected
|
|
request already carries it.
|
|
|
|
### Inspecting and clearing
|
|
|
|
#### `s:cookies()`
|
|
|
|
Return the unexpired cookies as a nested table, `{domain = {name = value}}`,
|
|
for inspection.
|
|
|
|
```lua
|
|
local jar = s:cookies()
|
|
for domain, names in pairs(jar) do
|
|
for name, value in pairs(names) do
|
|
print(domain, name, value)
|
|
end
|
|
end
|
|
```
|
|
|
|
#### `s:clearCookies()`
|
|
|
|
Empty the in-memory jar.
|
|
|
|
### Persisting the jar
|
|
|
|
Cookies live in memory for the session's lifetime. Save them to reuse a logged-in
|
|
session across script runs.
|
|
|
|
Jar files hold live credentials, so both directions run the
|
|
[fs permission system](fs.md#permissions) — write for `s:save`, read for
|
|
`s:load` and `http.session(path)`. Note also that `--sandbox` disables
|
|
networking entirely: every request errors with
|
|
`http: networking disabled by --sandbox`.
|
|
|
|
#### `s:save(path)`
|
|
|
|
Write the jar to `path` as JSONL — one JSON object per line, one cookie per
|
|
line, in `cookie_store`'s format (the full cookie: name, value, domain, path,
|
|
expiry). Session cookies — ones without an `Expires`/`Max-Age`, which is what
|
|
most login tokens are — are included.
|
|
|
|
#### `s:load(path)`
|
|
|
|
Load a jar file written by `s:save`, **replacing** the current jar contents.
|
|
`http.session(path)` is the same as creating a session and calling
|
|
`:load(path)`. A file that is not a saved jar raises an error.
|
|
|
|
```lua
|
|
-- First run: log in and persist
|
|
local s = http.session()
|
|
s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
|
|
s:save("session.jsonl")
|
|
|
|
-- Later run: restore and continue without logging in again
|
|
local s = http.session("session.jsonl")
|
|
local page = s:get("https://example.com/dashboard")
|
|
```
|
|
|
|
## Redirects
|
|
|
|
Redirects are followed automatically, up to 10 hops; you receive the final
|
|
response (`resp.url` tells you where you ended up). A `303`, and a `301`/`302`
|
|
in response to a `POST`, are followed as a bodyless `GET`, matching browser
|
|
behaviour. For a session, cookies set along the way are captured at each hop.
|
|
|
|
When a redirect leaves the original host, the `Authorization` and `Cookie`
|
|
headers are dropped so credentials never reach a host the request was not
|
|
addressed to. Other custom headers follow the redirect, as in browsers and curl.
|
|
|
|
## Errors
|
|
|
|
Failing to obtain a response raises a Lua error: DNS failure, connection refused,
|
|
a TLS problem, or a timeout. An HTTP error *status* (4xx/5xx) does not — it is
|
|
reported through `resp.status`/`resp.ok`. Parsing a non-JSON body with
|
|
`resp.json()` also raises.
|
|
|
|
Wrap calls in `pcall` or `utils.try` where you want to handle failure
|
|
rather than abort:
|
|
|
|
```lua
|
|
local resp, err = utils.try(function()
|
|
return http.get("https://does-not-exist.invalid", { timeout = 5 })
|
|
end)
|
|
if not resp then
|
|
log.error("request failed: " .. tostring(err))
|
|
end
|
|
```
|
|
|
|
## Notes
|
|
|
|
- **HTTPS needs no setup.** The TLS stack (rustls + `ring`) is compiled in; trust
|
|
roots come from the system certificate store.
|
|
- **A default `User-Agent` is sent** (`a/<version>`) because some servers reject
|
|
requests without one. Override it with a `User-Agent` entry in `opts.headers`.
|
|
- **Requests don't block the event loop.** Network I/O runs on the async core, so
|
|
a slow request does not stall other async work (timers, `os.sleep`, SQLite) in
|
|
the same script.
|
|
- **The cookie jar is per session.** Plain `http.get`/`http.post` calls do not
|
|
retain cookies between calls; use a session for that.
|
|
|
|
## Full example
|
|
|
|
```lua
|
|
-- Talk to a JSON API with a bearer token, then drive a stateful session.
|
|
|
|
-- One-off authenticated JSON call
|
|
local items, resp = http.getJSON("https://api.example.com/items", {
|
|
headers = { ["Authorization"] = "Bearer " .. token },
|
|
timeout = 15,
|
|
})
|
|
if not resp.ok then
|
|
error("list failed: HTTP " .. resp.status)
|
|
end
|
|
for _, item in ipairs(items) do
|
|
print(item.id, item.name)
|
|
end
|
|
|
|
-- Create one
|
|
local created = http.postJSON("https://api.example.com/items", { name = "widget" }, {
|
|
headers = { ["Authorization"] = "Bearer " .. token },
|
|
})
|
|
print("created id:", created.json().id)
|
|
|
|
-- A login session that persists across runs
|
|
local s = http.session("session.jsonl") -- restore if present
|
|
local home = s:get("https://example.com/dashboard")
|
|
if home.status == 401 then -- session expired; log in again
|
|
s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
|
|
home = s:get("https://example.com/dashboard")
|
|
s:save("session.jsonl")
|
|
end
|
|
print(home.ok and "logged in" or "login failed")
|
|
```
|
|
|