Skip to content

Analytics Section Migration #451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Ethan-Arrowood wants to merge 3 commits into
base: major-version-reorg
Choose a base branch
from
Open

Analytics Section Migration #451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
13d63b4
docs: migrate Analytics section to v4 consolidated reference
Ethan-Arrowood Mar 3, 2026
0f22de4
fixup! docs: migrate Analytics section to v4 consolidated reference
Ethan-Arrowood Mar 3, 2026
bb0310c
Merge remote-tracking branch 'origin/major-version-reorg' into migrat…
Ethan-Arrowood Mar 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions migration-context/link-placeholders/analytics-link-placeholders.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Link Placeholders for Analytics Section

## reference_versioned_docs/version-v4/analytics/overview.md

- Line 99: `[server.recordAnalytics()](TODO:reference_versioned_docs/version-v4/http/api.md)`
- Context: Noting that applications can record custom metrics via this API
- Target should be: HTTP API page, `server.recordAnalytics` section (file already exists)

- Line 103: `[HTTP API](TODO:reference_versioned_docs/version-v4/http/api.md)`
- Context: Related section — custom metrics API
- Target should be: HTTP API page (file already exists)

- Line 105: `[analytics.logging](TODO:reference_versioned_docs/version-v4/logging/configuration.md)`
- Context: Per-component analytics logging configuration
- Target should be: Logging configuration page (already migrated in PR #450)

- Line 106: `[Configuration Overview](TODO:reference_versioned_docs/version-v4/configuration/overview.md)`
- Context: analytics.aggregatePeriod configuration
- Target should be: Configuration section overview page

- Line 110: `[HTTP API](TODO:reference_versioned_docs/version-v4/http/api.md)`
- Context: Related section at bottom of file
- Target should be: HTTP API page (file already exists)

- Line 111: `[Logging Configuration](TODO:reference_versioned_docs/version-v4/logging/configuration.md)`
- Context: Related section at bottom of file
- Target should be: Logging configuration page (already migrated in PR #450)

- Line 112: `[Configuration Overview](TODO:reference_versioned_docs/version-v4/configuration/overview.md)`
- Context: Related section at bottom of file
- Target should be: Configuration section overview page

## reference_versioned_docs/version-v4/analytics/operations.md

- Line 56: `[search_by_conditions](TODO:reference_versioned_docs/version-v4/operations-api/operations.md)`
- Context: Note that `conditions` parameter uses the same format as search_by_conditions
- Target should be: Operations API operations page

- Line 77: `[Operations API Overview](TODO:reference_versioned_docs/version-v4/operations-api/overview.md)`
- Context: Related section at bottom of file
- Target should be: Operations API section overview page
136 changes: 136 additions & 0 deletions reference_versioned_docs/version-v4/analytics/operations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
---
id: operations
title: Analytics Operations
---

<!-- Source: versioned_docs/version-4.7/developers/operations-api/analytics.md (primary) -->
<!-- Source: versioned_docs/version-4.7/reference/analytics.md (metric names and descriptions) -->
<!-- Source: release-notes/v4-tucker/4.7.0.md (confirmed new analytics/licensing functionality) -->

Operations for querying Harper analytics data. All operations require `superuser` permission.

Analytics data can also be queried directly via `search_by_conditions` on the `hdb_raw_analytics` and `hdb_analytics` tables in the `system` database — see [Analytics Overview](./overview) for details on the table structure.

---

## `list_metrics`

Returns the list of available metric names that can be queried with `get_analytics`.

### Parameters

| Parameter | Required | Type | Description |
| -------------- | -------- | -------- | ------------------------------------------------------------------------ |
| `operation` | Yes | string | Must be `"list_metrics"` |
| `metric_types` | No | string[] | Filter by type: `"builtin"`, `"custom"`, or both. Default: `["builtin"]` |

### Request

```json
{
"operation": "list_metrics",
"metric_types": ["custom", "builtin"]
}
```

### Response

```json
["resource-usage", "table-size", "database-size", "main-thread-utilization", "utilization", "storage-volume"]
```

---

## `describe_metric`

Returns the structure and available attributes for a specific metric.

### Parameters

| Parameter | Required | Type | Description |
| ----------- | -------- | ------ | ------------------------------ |
| `operation` | Yes | string | Must be `"describe_metric"` |
| `metric` | Yes | string | Name of the metric to describe |

### Request

```json
{
"operation": "describe_metric",
"metric": "resource-usage"
}
```

### Response

```json
{
"attributes": [
{ "name": "id", "type": "number" },
{ "name": "metric", "type": "string" },
{ "name": "userCPUTime", "type": "number" },
{ "name": "systemCPUTime", "type": "number" },
{ "name": "node", "type": "string" }
]
}
```

---

## `get_analytics`

Queries analytics data for a specific metric over a time range.

### Parameters

| Parameter | Required | Type | Description |
| ---------------- | -------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `operation` | Yes | string | Must be `"get_analytics"` |
| `metric` | Yes | string | Metric name — use `list_metrics` to get valid values |
| `start_time` | No | number | Start of time range as Unix timestamp in milliseconds |
| `end_time` | No | number | End of time range as Unix timestamp in milliseconds |
| `get_attributes` | No | string[] | Attributes to include in each result. If omitted, all attributes are returned |
| `conditions` | No | object[] | Additional filter conditions. Same format as [`search_by_conditions`](TODO:reference_versioned_docs/version-v4/operations-api/operations.md 'Operations API — search_by_conditions') |

### Request

```json
{
"operation": "get_analytics",
"metric": "resource-usage",
"start_time": 1769198332754,
"end_time": 1769198532754,
"get_attributes": ["id", "metric", "userCPUTime", "systemCPUTime"],
"conditions": [
{
"attribute": "node",
"operator": "equals",
"value": "node1.example.com"
}
]
}
```

### Response

```json
[
{
"id": "12345",
"metric": "resource-usage",
"userCPUTime": 100,
"systemCPUTime": 50
},
{
"id": "67890",
"metric": "resource-usage",
"userCPUTime": 150,
"systemCPUTime": 75
}
]
```

## Related

- [Analytics Overview](./overview)
- [Operations API Overview](TODO:reference_versioned_docs/version-v4/operations-api/overview.md 'Full Operations API reference')
207 changes: 207 additions & 0 deletions reference_versioned_docs/version-v4/analytics/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
---
id: overview
title: Analytics
---

<!-- Source: versioned_docs/version-4.7/reference/analytics.md (primary) -->
<!-- Source: release-notes/v4-tucker/4.5.0.md (confirmed resource and storage analytics added) -->
<!-- Source: release-notes/v4-tucker/4.7.0.md (confirmed new analytics/licensing functionality) -->

Added in: v4.5.0 (resource and storage analytics)

Harper collects real-time telemetry and statistics across all operations, URL endpoints, and messaging topics. This data can be used to monitor server health, understand traffic and usage patterns, identify resource-intensive queries, and inform scaling decisions.

## Storage Tables

Analytics data is stored in two system tables in the `system` database:

| Table | Description |
| ------------------- | ------------------------------------------------------------------------------------------- |
| `hdb_raw_analytics` | Per-second raw entries recorded by each thread. One record per second per active thread. |
| `hdb_analytics` | Aggregate entries recorded once per minute, summarizing all per-second data across threads. |

Both tables require `superuser` permission to query.

## Raw Analytics (`hdb_raw_analytics`)

Raw entries are recorded once per second (when there is activity) by each thread. Each record captures all activity in the last second along with system resource information. Records use the timestamp in milliseconds since epoch as the primary key.

Query raw analytics using `search_by_conditions` on the `hdb_raw_analytics` table. The example below fetches 10 seconds of raw entries:

```http
POST http://localhost:9925
Content-Type: application/json

{
"operation": "search_by_conditions",
"schema": "system",
"table": "hdb_raw_analytics",
"conditions": [{
"search_attribute": "id",
"search_type": "between",
"search_value": [1688594000000, 1688594010000]
}]
}
```

Example raw entry:

```json
{
"time": 1688594390708,
"period": 1000.8336279988289,
"metrics": [
{
"metric": "bytes-sent",
"path": "search_by_conditions",
"type": "operation",
"median": 202,
"mean": 202,
"p95": 202,
"p90": 202,
"count": 1
},
{
"metric": "memory",
"threadId": 2,
"rss": 1492664320,
"heapTotal": 124596224,
"heapUsed": 119563120,
"external": 3469790,
"arrayBuffers": 798721
},
{
"metric": "utilization",
"idle": 138227.52767700003,
"active": 70.5066209952347,
"utilization": 0.0005098165086230495
}
],
"threadId": 2,
"totalBytesProcessed": 12182820,
"id": 1688594390708.6853
}
```

## Aggregate Analytics (`hdb_analytics`)

Aggregate entries are recorded once per minute, combining per-second raw entries from all threads into a single summary record. Use `search_by_conditions` on the `hdb_analytics` table with a broader time range:

```http
POST http://localhost:9925
Content-Type: application/json

{
"operation": "search_by_conditions",
"schema": "system",
"table": "hdb_analytics",
"conditions": [{
"search_attribute": "id",
"search_type": "between",
"search_value": [1688194100000, 1688594990000]
}]
}
```

Example aggregate entry:

```json
{
"period": 60000,
"metric": "bytes-sent",
"method": "connack",
"type": "mqtt",
"median": 4,
"mean": 4,
"p95": 4,
"p90": 4,
"count": 1,
"id": 1688589569646,
"time": 1688589569646
}
```

## Standard Metrics

Harper automatically tracks the following metrics for all services. Applications can also define custom metrics via [`server.recordAnalytics()`](TODO:reference_versioned_docs/version-v4/http/api.md 'HTTP API — server.recordAnalytics').

### HTTP Metrics

| `metric` | `path` | `method` | `type` | Unit | Description |
| ------------------ | ------------- | -------------- | --------------------------- | ----- | ---------------------------------------- |
| `duration` | resource path | request method | `cache-hit` or `cache-miss` | ms | Duration of request handler |
| `duration` | route path | request method | `fastify-route` | ms | Duration of Fastify route handler |
| `duration` | operation | | `operation` | ms | Duration of Operations API operation |
| `success` | resource path | request method | | % | Percentage of successful requests |
| `success` | route path | request method | `fastify-route` | % | |
| `success` | operation | | `operation` | % | |
| `bytes-sent` | resource path | request method | | bytes | Response bytes sent |
| `bytes-sent` | route path | request method | `fastify-route` | bytes | |
| `bytes-sent` | operation | | `operation` | bytes | |
| `transfer` | resource path | request method | `operation` | ms | Duration of response transfer |
| `transfer` | route path | request method | `fastify-route` | ms | |
| `transfer` | operation | | `operation` | ms | |
| `socket-routed` | | | | % | Percentage of sockets immediately routed |
| `tls-handshake` | | | | ms | TLS handshake duration |
| `tls-reused` | | | | % | Percentage of TLS sessions reused |
| `cache-hit` | table name | | | % | Percentage of cache hits |
| `cache-resolution` | table name | | | ms | Duration of resolving uncached entries |

### MQTT / WebSocket Metrics

| `metric` | `path` | `method` | `type` | Unit | Description |
| ------------------ | ------ | ------------ | ------ | ----- | ------------------------------------------------ |
| `mqtt-connections` | | | | count | Number of open direct MQTT connections |
| `ws-connections` | | | | count | Number of open WebSocket connections |
| `connection` | `mqtt` | `connect` | | % | Percentage of successful direct MQTT connections |
| `connection` | `mqtt` | `disconnect` | | % | Percentage of explicit direct MQTT disconnects |
| `connection` | `ws` | `connect` | | % | Percentage of successful WebSocket connections |
| `connection` | `ws` | `disconnect` | | % | Percentage of explicit WebSocket disconnects |
| `bytes-sent` | topic | mqtt command | `mqtt` | bytes | Bytes sent for a given MQTT command and topic |

### Replication Metrics

| `metric` | `path` | `method` | `type` | Unit | Description |
| ---------------- | ------------- | ------------- | --------- | ----- | ----------------------------------- |
| `bytes-sent` | node.database | `replication` | `egress` | bytes | Bytes sent for replication |
| `bytes-sent` | node.database | `replication` | `blob` | bytes | Bytes sent for blob replication |
| `bytes-received` | node.database | `replication` | `ingress` | bytes | Bytes received for replication |
| `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication |

### Resource Usage Metrics

| `metric` | Key attributes | Other | Unit | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------- |
| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown |
| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown |
| `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) |
| `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown |
| `table-size` | `size` | `database`, `table` | bytes | Table file size |
| `utilization` | | | % | Percentage of time the worker thread was processing requests |

#### `resource-usage` Metric

Includes everything returned by Node.js [`process.resourceUsage()`](https://nodejs.org/api/process.html#processresourceusage) (with `userCPUTime` and `systemCPUTime` converted to milliseconds), plus:

| Field | Unit | Description |
| ---------------- | ---- | ------------------------------------------- |
| `time` | ms | Unix timestamp when the metric was recorded |
| `period` | ms | Duration of the measurement period |
| `cpuUtilization` | % | CPU utilization (user + system combined) |

## Custom Metrics

Applications can record custom metrics using the `server.recordAnalytics()` API. See [HTTP API](TODO:reference_versioned_docs/version-v4/http/api.md 'server.recordAnalytics API') for details.

## Analytics Configuration

The `analytics.aggregatePeriod` configuration option controls how frequently aggregate summaries are written. See [Configuration Overview](TODO:reference_versioned_docs/version-v4/configuration/overview.md 'Full harperdb-config.yaml reference') for details.

Per-component analytics logging can be configured via `analytics.logging`. See [Logging Configuration](TODO:reference_versioned_docs/version-v4/logging/configuration.md 'analytics.logging configuration').

## Related

- [Analytics Operations](./operations)
- [HTTP API](TODO:reference_versioned_docs/version-v4/http/api.md 'server.recordAnalytics')
- [Logging Configuration](TODO:reference_versioned_docs/version-v4/logging/configuration.md 'analytics.logging')
- [Configuration Overview](TODO:reference_versioned_docs/version-v4/configuration/overview.md 'Full configuration reference')
Loading