You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
354 lines
12 KiB
354 lines
12 KiB
# `http`
|
|
|
|
The `http` module is an HTTP/HTTPS client for calling APIs, scraping pages, and
|
|
submitting forms. TLS is compiled into the binary (rustls with the `ring`
|
|
provider), so HTTPS works with no system library. There are two ways in: plain
|
|
functions for one-off requests, and a **session** object that carries a cookie
|
|
jar across requests for stateful flows like logging in.
|
|
|
|
```lua
|
|
local resp = http.get("https://example.com")
|
|
print(resp.status, resp.ok) --> 200 true
|
|
print(resp.body) --> "<!doctype html>…"
|
|
|
|
-- JSON in one step
|
|
local data = http.getJSON("https://api.example.com/users")
|
|
for _, user in ipairs(data) do
|
|
print(user.name)
|
|
end
|
|
```
|
|
|
|
## Making requests
|
|
|
|
Every helper takes a URL and an optional `opts` table (see [Request
|
|
options](#request-options)) and returns a [response](#the-response) table.
|
|
|
|
```lua
|
|
http.get(url [, opts])
|
|
http.post(url [, opts])
|
|
http.put(url [, opts])
|
|
http.patch(url [, opts])
|
|
http.delete(url [, opts])
|
|
http.head(url [, opts])
|
|
```
|
|
|
|
These are thin shorthands over the one primitive:
|
|
|
|
### `http.request(method, url [, opts])`
|
|
|
|
Send a request with an explicit method (any verb, e.g. `"GET"`, `"OPTIONS"`) and
|
|
return the response. The shorthands above are just `http.request` with the method
|
|
filled in.
|
|
|
|
```lua
|
|
local resp = http.request("DELETE", "https://api.example.com/items/42")
|
|
```
|
|
|
|
## The response
|
|
|
|
Every request returns a table describing the response:
|
|
|
|
| Field | Type | Description |
|
|
|-----------|-----------|-------------------------------------------------------------------|
|
|
| `status` | integer | HTTP status code, e.g. `200`, `404`. |
|
|
| `ok` | boolean | `true` when `status` is in the 2xx range. |
|
|
| `headers` | table | Response headers, keyed by **lowercase** name. |
|
|
| `body` | string | The raw response body (Lua strings are byte sequences). |
|
|
| `json` | function | `resp.json()` parses `body` as JSON. See [JSON](#json). |
|
|
|
|
Header names are lowercased so you can look them up without guessing the server's
|
|
capitalization. If a header appears more than once, the first value wins.
|
|
|
|
```lua
|
|
local resp = http.get("https://example.com")
|
|
print(resp.headers["content-type"]) --> "text/html; charset=utf-8"
|
|
```
|
|
|
|
A non-2xx status is **not** an error — `resp.status` and `resp.ok` simply report
|
|
it. Only a failure to get a response at all (DNS, connection, TLS, timeout) raises
|
|
a Lua error. See [Errors](#errors).
|
|
|
|
## Request options
|
|
|
|
The optional `opts` table accepts these fields, all optional:
|
|
|
|
| Field | Type | Behaviour |
|
|
|-----------|--------|--------------------------------------------------------------------------|
|
|
| `headers` | table | Extra request headers, `{["X-Foo"] = "bar"}`. |
|
|
| `body` | string | Raw request body; set `Content-Type` yourself via `headers`. |
|
|
| `json` | any | Serialized to JSON; sets `Content-Type: application/json`. |
|
|
| `form` | table | URL-encoded; sets `Content-Type: application/x-www-form-urlencoded`. |
|
|
| `cookies` | table | Cookies for this request, `{session = "abc"}` → `Cookie:` header. |
|
|
| `timeout` | number | Per-request timeout in seconds (default `30`). |
|
|
| `auth` | table | Basic or digest credentials. See [Authentication](#authentication). |
|
|
|
|
### Bodies
|
|
|
|
`json`, `form`, and `body` are three ways to set the request body; if more than
|
|
one is given, the first present in that order wins.
|
|
|
|
```lua
|
|
-- JSON body (table serialized to an object)
|
|
http.post("https://api.example.com/items", { json = { name = "test", count = 5 } })
|
|
|
|
-- Form submission (application/x-www-form-urlencoded)
|
|
http.post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
|
|
|
|
-- Raw body with an explicit content type
|
|
http.post("https://example.com/ingest", {
|
|
body = "id,name\n1,alice\n",
|
|
headers = { ["Content-Type"] = "text/csv" },
|
|
})
|
|
```
|
|
|
|
`json` serializes the same way as `utils.toJSON`: a table with
|
|
sequential integer keys becomes a JSON array, otherwise an object, and
|
|
`utils.NULL` becomes JSON `null` (a literal Lua `nil` cannot live in
|
|
a table).
|
|
|
|
### Headers and cookies
|
|
|
|
```lua
|
|
local resp = http.get("https://example.com", {
|
|
headers = { ["Accept"] = "application/json", ["X-Token"] = "xyz" },
|
|
cookies = { session = "abc123" },
|
|
timeout = 10,
|
|
})
|
|
```
|
|
|
|
Per-request `cookies` are sent as a `Cookie` header. With a [session](#sessions),
|
|
they are merged with the jar and take precedence on a name collision.
|
|
|
|
## JSON
|
|
|
|
### `resp.json()`
|
|
|
|
Parse the response body as JSON and return the resulting Lua value. It is the
|
|
counterpart of `utils.fromJSON`: JSON objects become tables, arrays
|
|
become array-tables, and `null` becomes `utils.NULL`. Calling it on a
|
|
body that is not valid JSON raises an error.
|
|
|
|
```lua
|
|
local resp = http.post("https://api.example.com/echo", { json = { hello = "world" } })
|
|
print(resp.json().hello) --> "world"
|
|
```
|
|
|
|
### `http.getJSON(url [, opts])`
|
|
|
|
Shorthand for a GET that parses the body. Returns **two** values: the parsed body
|
|
and the full response.
|
|
|
|
```lua
|
|
local data, resp = http.getJSON("https://api.example.com/users")
|
|
print(resp.status, #data)
|
|
```
|
|
|
|
### `http.postJSON(url, body [, opts])`
|
|
|
|
Shorthand for a POST with a JSON body — equivalent to setting `opts.json = body`.
|
|
Returns the response.
|
|
|
|
```lua
|
|
local resp = http.postJSON("https://api.example.com/items", { name = "test", count = 5 })
|
|
if resp.ok then print(resp.json().id) end
|
|
```
|
|
|
|
## Authentication
|
|
|
|
Pass credentials in `opts.auth`. The `scheme` is `"basic"` (the default) or
|
|
`"digest"`.
|
|
|
|
| Field | Type | Description |
|
|
|------------|--------|----------------------------------------------|
|
|
| `username` | string | Required. |
|
|
| `password` | string | Required. |
|
|
| `scheme` | string | `"basic"` (default) or `"digest"`. |
|
|
|
|
```lua
|
|
-- HTTP Basic
|
|
local resp = http.get("https://api.example.com/private", {
|
|
auth = { username = "alice", password = "secret" },
|
|
})
|
|
|
|
-- HTTP Digest — the 401 challenge is answered automatically
|
|
local resp = http.get("https://api.example.com/private", {
|
|
auth = { username = "alice", password = "secret", scheme = "digest" },
|
|
})
|
|
```
|
|
|
|
For basic auth the `Authorization` header is sent with the request. For digest the
|
|
client sends the request, reads the server's `401` challenge, computes the
|
|
response, and retries once; you only see the final response. Auth works the same
|
|
way on [sessions](#sessions).
|
|
|
|
## Sessions
|
|
|
|
A **session** wraps its own cookie jar. Cookies from `Set-Cookie` responses are
|
|
stored automatically and sent back on later requests to matching hosts — which is
|
|
what makes login-then-fetch flows work. A session also has the same request
|
|
methods as the plain module.
|
|
|
|
### `http.session([path])`
|
|
|
|
Create a session. With a `path`, the cookie jar is preloaded from that file (see
|
|
[Persisting the jar](#persisting-the-jar)).
|
|
|
|
```lua
|
|
local s = http.session() -- fresh, empty jar
|
|
local s = http.session("cookies.jsonl") -- jar loaded from disk
|
|
```
|
|
|
|
### Requests
|
|
|
|
A session has every method the plain module has — `request`, `get`, `post`,
|
|
`put`, `patch`, `delete`, `head`, `getJSON`, `postJSON` — called with `:` syntax
|
|
and accepting the same `opts`:
|
|
|
|
```lua
|
|
local s = http.session()
|
|
|
|
-- Log in; the response's Set-Cookie is captured into the jar
|
|
s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
|
|
|
|
-- The session cookie is sent automatically
|
|
local page = s:get("https://example.com/dashboard")
|
|
```
|
|
|
|
### Cookie behaviour
|
|
|
|
A stored cookie is sent on a request when the request host **equals** the cookie's
|
|
domain or is a subdomain of it. The jar reads the `name=value` pair and the
|
|
`Domain` attribute from each `Set-Cookie` header; other attributes (`Path`,
|
|
`Expires`, `Secure`, `HttpOnly`) are ignored. When no `Domain` is given, the
|
|
request host is used.
|
|
|
|
`Set-Cookie` headers set on redirect responses are captured too, so a login that
|
|
`302`-redirects to a dashboard still records its cookie.
|
|
|
|
### Inspecting and clearing
|
|
|
|
#### `s:cookies()`
|
|
|
|
Return the jar as a nested table, `{domain = {name = value}}`, for inspection.
|
|
|
|
```lua
|
|
local jar = s:cookies()
|
|
for domain, names in pairs(jar) do
|
|
for name, value in pairs(names) do
|
|
print(domain, name, value)
|
|
end
|
|
end
|
|
```
|
|
|
|
#### `s:clearCookies()`
|
|
|
|
Empty the in-memory jar.
|
|
|
|
### Persisting the jar
|
|
|
|
Cookies live in memory for the session's lifetime. Save them to reuse a logged-in
|
|
session across script runs.
|
|
|
|
Jar files hold live credentials, so both directions run the
|
|
[fs permission system](fs.md#permissions) — write for `s:save`, read for
|
|
`s:load` and `http.session(path)`. Note also that `--sandbox` disables
|
|
networking entirely: every request errors with
|
|
`http: networking disabled by --sandbox`.
|
|
|
|
#### `s:save(path)`
|
|
|
|
Write the jar to `path` as JSONL — one JSON object per line, one cookie per line:
|
|
|
|
```jsonl
|
|
{"domain":"example.com","name":"session","value":"abc123"}
|
|
{"domain":"api.example.com","name":"token","value":"xyz789"}
|
|
```
|
|
|
|
#### `s:load(path)`
|
|
|
|
Merge cookies from a JSONL file into the current jar (existing cookies are kept,
|
|
matching names overwritten). `http.session(path)` is the same as creating a
|
|
session and calling `:load(path)`.
|
|
|
|
```lua
|
|
-- First run: log in and persist
|
|
local s = http.session()
|
|
s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
|
|
s:save("session.jsonl")
|
|
|
|
-- Later run: restore and continue without logging in again
|
|
local s = http.session("session.jsonl")
|
|
local page = s:get("https://example.com/dashboard")
|
|
```
|
|
|
|
## Redirects
|
|
|
|
Redirects are followed automatically, up to 10 hops; you receive the final
|
|
response. A `303`, and a `301`/`302` in response to a `POST`, are followed as a
|
|
bodyless `GET`, matching browser behaviour. For a session, cookies set along the
|
|
way are captured at each hop.
|
|
|
|
## Errors
|
|
|
|
Failing to obtain a response raises a Lua error: DNS failure, connection refused,
|
|
a TLS problem, or a timeout. An HTTP error *status* (4xx/5xx) does not — it is
|
|
reported through `resp.status`/`resp.ok`. Parsing a non-JSON body with
|
|
`resp.json()` also raises.
|
|
|
|
Wrap calls in `pcall` or `utils.try` where you want to handle failure
|
|
rather than abort:
|
|
|
|
```lua
|
|
local resp, err = utils.try(function()
|
|
return http.get("https://does-not-exist.invalid", { timeout = 5 })
|
|
end)
|
|
if not resp then
|
|
log.error("request failed: " .. tostring(err))
|
|
end
|
|
```
|
|
|
|
## Notes
|
|
|
|
- **HTTPS needs no setup.** The TLS stack (rustls + `ring`) is compiled in; trust
|
|
roots come from the system certificate store.
|
|
- **A default `User-Agent` is sent** (`a/<version>`) because some servers reject
|
|
requests without one. Override it with a `User-Agent` entry in `opts.headers`.
|
|
- **Requests don't block the event loop.** Network I/O runs on the async core, so
|
|
a slow request does not stall other async work (timers, `os.sleep`, SQLite) in
|
|
the same script.
|
|
- **The cookie jar is per session.** Plain `http.get`/`http.post` calls do not
|
|
retain cookies between calls; use a session for that.
|
|
|
|
## Full example
|
|
|
|
```lua
|
|
-- Talk to a JSON API with a bearer token, then drive a stateful session.
|
|
|
|
-- One-off authenticated JSON call
|
|
local items, resp = http.getJSON("https://api.example.com/items", {
|
|
headers = { ["Authorization"] = "Bearer " .. token },
|
|
timeout = 15,
|
|
})
|
|
if not resp.ok then
|
|
error("list failed: HTTP " .. resp.status)
|
|
end
|
|
for _, item in ipairs(items) do
|
|
print(item.id, item.name)
|
|
end
|
|
|
|
-- Create one
|
|
local created = http.postJSON("https://api.example.com/items", { name = "widget" }, {
|
|
headers = { ["Authorization"] = "Bearer " .. token },
|
|
})
|
|
print("created id:", created.json().id)
|
|
|
|
-- A login session that persists across runs
|
|
local s = http.session("session.jsonl") -- restore if present
|
|
local home = s:get("https://example.com/dashboard")
|
|
if home.status == 401 then -- session expired; log in again
|
|
s:post("https://example.com/login", { form = { user = "alice", pass = "secret" } })
|
|
home = s:get("https://example.com/dashboard")
|
|
s:save("session.jsonl")
|
|
end
|
|
print(home.ok and "logged in" or "login failed")
|
|
```
|
|
|