Debug and monitor
Every realtime server has a live view of its traffic, lookups for channels and connections, and per-app insights. These views show end-user data, so they need the Realtime publish permission; the summaries need only Realtime read.
Debug console and live tail
The Debug console shows connections, subscriptions, messages, API calls, errors and webhook attempts as they happen. You can filter by app, event family and channel prefix, send test events and inspect a channel. The CLI streams the same events:
hubfly realtime tail chat
hubfly realtime tail chat --app production --family msg,conn --channel-prefix orders
hubfly realtime tail chat --json | jq .| Family | Events |
|---|---|
conn | Connections opening, closing (with close code), detaching, resuming and signing in, with IP, origin and client library |
sub | Subscribes, unsubscribes and subscription errors; channels becoming occupied or vacated |
member | Presence members added and removed |
msg | Publishes, sends and client events: channels, event name, size, recipients, source and whether it was encrypted |
api | Server API requests: method, path without the query string, status and duration |
err | Authorization failures, rate limits, encryption policy refusals and protocol errors |
webhook | Webhook delivery attempts and their results |
- Payloads are hidden. To see the first 1 KiB of messages, turn on Payload preview in the debug console in the app settings and ask for previews (
--previewin the CLI). Encrypted payloads are never previewed, and their event names show ashubfly.e2e. - Sampling. Above 500 events per second the view samples and says so. At most 5 debug sessions can be open per server, each for up to an hour; the CLI reconnects with a new session when one ends.
- The console can decrypt an encrypted payload locally in your browser if you paste the key; the key is never sent anywhere.
Channels, connections and history
hubfly realtime channels chat --prefix presence-
hubfly realtime channels inspect chat presence-room-42 --members 20
hubfly realtime connections chat --user u_42
hubfly realtime disconnect chat --user u_42 # closes the user's connections
hubfly realtime history chat --channel orders --limit 20
hubfly realtime history purge chat --channel orders # deletes stored messages; asks firstThe Dashboard's History section looks up a channel's stored messages with their epoch and offset, and can purge them. After a purge, clients that reconnect cannot catch up on the deleted messages.
Insights and usage
Insights show, per app, connections, message and bandwidth rates, top channels and top events, connection closes by close code, client libraries, authorization failures, rate-limit hits, webhook success and backlog, and daily usage.
hubfly realtime insights chat --window 6h
hubfly realtime usage chat --since 168hLive insights are kept in the server's memory for 24 hours, so they start again from zero after a restart, update or crash. Daily usage is stored on the server's volume.
Logs, metrics and alerts
hubfly realtime logs chat --follow
hubfly realtime metrics chat --window 1h
hubfly realtime operations chatAlerts notify you by email when connections approach the size's limit, when the server is down, and when webhooks keep failing. The Activity section lists operations and audited actions such as key reveals, publishes and disconnects.
Troubleshooting
| Symptom | Likely cause |
|---|---|
| Clients close with 4001 | The app was deleted, or the key was revoked or its rotation grace period ended. |
| Clients close with 4003 | The app is disabled. |
| Clients close with 4009 | The page's origin is not in the app's allowed origins, user sign-in failed or timed out, or the connection was terminated. |
| Clients close with 4100 | Over capacity: the server's or app's connection limit, a client reading too slowly, or the bandwidth budget used up. Resize, or reduce fan-out. |
pusher:subscription_error with 401 or 403 | Your auth endpoint refused or signed with another app's secret; or a user channel of someone else, or unsealed presence data under the required policy. |
| Server API answers 403 before your request reaches the app | Your server is not on the server API allowlist (WebSocket clients are not affected). |
| 413 on publish or send | The payload is above the message limit of the size or app. |
403 with reason e2e_required | The app's encryption policy is required and the payload is plaintext. |
A user message reports offline although the user is connected | The connection signed in but did not join the user's channel. pusher-js and the Hubfly space client do this automatically after sign-in. |
Hubfly space client logs e2e:error with reason plaintext | Something published plaintext into an encrypted context. Encrypt on your server. |
| A webhook is never delivered | The URL resolves to a refused address range or uses a port other than 443 or 8443. Use Test and the delivery log. |
More detail