# CT Intelligence Service — full specification (2026-05-30)

**Status: DESIGN / build-ready.** The "brain" that decides *who CT copies, how
much, how it exits, and how it lands* — a standalone service separate from the
trading bot. It never trades (no wallet, no keys, no send path). It reads the
chain, ranks wallets, and writes one config file the bot consumes.

> **Two processes, one file.** `ct-intel` (this service) thinks; `ct-bot` (the
> existing trading bot) acts. They communicate only through
> `leader-intelligence.json`. `ct-intel` runs 24/7 even when `ct-bot` is off.

---

## 0. Components at a glance

```
 Yellowstone gRPC ─► ct-indexer ─► ClickHouse ─► scorer ─► leader-intelligence.json ─► rsync ─► ct-bot
   (already paid)     (decode)     (30d data)   (rank)        (the config)            (hosts)   (reads it)
                                                   ▲                                              │
                                                   └──────────── feedback (CT fills/landings) ◄────┘
```

| # | Component | What it is | Reuses |
|---|---|---|---|
| 1 | **ct-indexer** | gRPC consumer → DB writer, never trades | CT `ingest/corvus.rs` + `swap/` decode |
| 2 | **ClickHouse** | local 30-day rolling store | — (new, ~3 tables) |
| 3 | **scorer** | query suite → metrics → ranking → outputs | `build_dashboard.py` logic |
| 4 | **leader-intelligence.json** | the bot-readable config | spec §6 |
| 5 | **ct-bot reader** | `leader_intel.rs`, hot-reload, gated | CT `MonitorCtx`, `PER_WALLET_EXIT` |

---

## 1. ct-indexer — what to collect & how

A standalone Rust binary (`src/bin/ct-indexer.rs`), measurement-only (the L-1
never-dispatch property: zero `WorkItem`/dispatch/swap-send code). Subscribes to
Yellowstone gRPC for the **programs** (not a wallet list) so it sees the whole
population — required for discovery/harvest:

- pump.fun program · pumpswap (pump_amm) program.

For **every** decoded instruction it extracts and writes:

| Field | Source in the gRPC tx message | Used for |
|---|---|---|
| `signer` (wallet) | tx.message account_keys[0] | who traded |
| `mint` | decoded buy/sell ix accounts | the token |
| `side` (buy/sell) | which ix (buy vs sell) | direction |
| `sol_amount`, `token_amount` | ix args / balance deltas | size, price, PnL |
| `slot`, `block_time` | tx context | timing, windows |
| `fee` | tx.meta.fee | cost / breakeven |
| `compute_units` | tx.meta.compute_units_consumed | landing recipe (CU) |
| `tx_index` | tx.index (position in block) | **intra-slot index** |
| `is_durable_nonce` | `SysvarRecentBlockhashes` in account_keys | infra |
| `tip_account` | known tip pubkeys in account_keys | provider / tip |
| `success` | tx.meta.err is null | landing % |
| `max_sol_cost` / `min_sol_output` | buy/sell ix args | **slippage tolerance** |
| (create ix) `creator`, `genesis_slot` | pump create ix | **E9 insider gate** |

**Everything we pulled from Dune is in the stream** — `tx.meta` carries fee +
compute units + err; `tx.index` is the intra-slot position; the ix args carry
slippage; the create ix carries the creator. Nothing is lost.

Write path: batch-insert (e.g. 1–5 s flush) into ClickHouse. Backpressure-safe,
drop-log on overflow (it's analytics, not money). Runs under PM2, autorestart.

## 2. ClickHouse — how to store it

A modest box (4–8 GB RAM). Three append-only tables, **30-day TTL** so storage
stays bounded (pump.fun ≈ 5–15M trades/day ⇒ ~150–450M rows live; ClickHouse
handles this trivially with partitioning by day).

```sql
CREATE TABLE trades (                       -- one row per buy/sell
  ts DateTime, slot UInt64, wallet String, mint String,
  side Enum('b','s'), sol Float64, tok Float64,
  tx_id String, tx_index UInt32, fee UInt64, cu UInt32,
  durable_nonce UInt8, tip_provider LowCardinality(String), success UInt8,
  slip_bound Float64                         -- max_sol_cost (buy) / min_sol_output (sell)
) ENGINE = MergeTree ORDER BY (mint, slot)
  PARTITION BY toDate(ts) TTL ts + INTERVAL 30 DAY;

CREATE TABLE creates (                       -- one row per token launch
  ts DateTime, slot UInt64, mint String, creator String
) ENGINE = MergeTree ORDER BY mint
  PARTITION BY toDate(ts) TTL ts + INTERVAL 120 DAY;   -- longer: needed for the E9 gate

CREATE TABLE fills (                          -- OUR OWN trades, fed back by ct-bot (§7)
  ts DateTime, leader String, mint String, our_sol Float64,
  our_slot_delta Int32, our_cu UInt32, sell_reason String, realized_roi Float64
) ENGINE = MergeTree ORDER BY (leader, ts) PARTITION BY toDate(ts) TTL ts + INTERVAL 90 DAY;
```

WSOL pairing, mint↔creator, and price are all derivable from `trades`+`creates`.

## 3. scorer — the query suite (what to look for)

A scheduled job (Python, reuses `build_dashboard.py`). The DuneSQL we wrote this
session ports to ClickHouse ≈1:1 (window functions, `quantile()`, joins). It runs
the suite, computes per-wallet metrics, ranks, and emits the outputs.

| Query | Computes | Drives |
|---|---|---|
| **discovery** | per wallet: PnL/WR/tokens/hold/size (30d) | the candidate universe |
| **creator E9** | % of a wallet's buys ≤2 slots of genesis | insider gate (exclude) |
| **infra** | CU · fee · tip/provider · nonce% · success% · **intra-slot idx** | landing recipe + classify |
| **exit-recon** | entry/peak/exit → giveback (TSL) · hold · SL band | per-leader EXIT |
| **mirror** | copier-sell vs leader-sell gap | confirm own-exit |
| **leader-list** | who copies whom (reverse-upstream) | competitor set + harvest |
| **landing-recipe** | competitors' p25/50/75 CU/tip/Δslot/idx + winners | per-leader LANDING |
| **harvest** | recurring out-of-cohort upstream wallets | NEW alpha candidates (auto) |
| **slippage** | buy headroom / sell floor | entry/exit slippage config |

## 4. Scoring / derivation logic

**① Tier ladder (who + how much)** — event-driven, recoverable:
`BENCH(0,shadow) ⇄ PROBATION(.5) ⇄ SECONDARY(.75) ⇄ PRIMARY(1.0)`.
Inputs: rolling PnL · WR · selectivity · E9-clean · liveness · single-exit fit.
Rule: N losing windows demote 1 rung; N winning promote 1; strikes reset on move.
DON'T-copy only on bad **signal** (MM/wash/accumulator/unprofitable) — an
alpha's own landing is irrelevant (we land it our way).

**② Exit (when out)** — from exit-recon per leader:
`tp=0 (arm from buy) · tsl≈median giveback · sl≈|p25 exit-ROI| (12–28) ·
autosell≈1.5×p75 hold`. Mode = FLIP / TRAIL / SWING.

**③ Landing (how to sit ahead)** — from the landing recipe per leader:
`target Δslot = beat winners' p25 · cu = winners' p75 · tip = winners' p75 ·
lane = durable-nonce multi-relay`.

## 5. (covered above)

## 6. Output — leader-intelligence.json (the contract)

```json
{ "generated_at": 1780150000, "window_days": 30,
  "leaders": {
    "<leader_base58>": {
      "tier": "PRIMARY", "size_mult": 1.0,
      "exit":    { "mode":"TRAIL", "tp":0, "tsl":30, "sl":20, "autosell":130,
                   "ladder": {"enabled":false, "first_pct":30, "size_pct":34, "rungs":2} },
      "landing": { "target_dslot":4, "cu_price":107000, "tip_lamports":0,
                   "lane":"durable_nonce_multirelay" },
      "evidence":{ "pnl_30d":1847, "wr":62, "exit_n":2880, "competitors":1367, "updated":1780150000 }
    }
  }
}
```

## 7. ct-bot reader — how it's passed to CT

New module `src/leader_intel.rs`, gated `LEADER_INTEL_ENABLED` (default off ⇒
byte-identical to today). It maps each json field onto a knob CT **already has**:

| json field | CT mechanism (existing) |
|---|---|
| leader (tier ≥ PROBATION) | added to `ctx.wallets` (the tracked-signer DashSet) |
| `size_mult` | multiplies `BUY_SOL_AMOUNT` for that leader's copies |
| `exit{tp,tsl,sl,autosell}` | `ctx.per_wallet_exit` (S2 `ExitOverride` — already built) |
| `landing.cu_price` | `effective_cu_price()` for that leader's copy |
| `landing.tip_lamports` | tip for that leader's copy |
| `landing.lane` | send-lane selection |

Lifecycle:
1. **Boot**: parse the json → build the maps → populate `ctx.wallets`,
   `ctx.per_wallet_exit`, `ctx.per_wallet_landing` (new field).
2. **Hot-reload**: a 30-min timer re-reads the file (atomic map swap). A demoted
   leader leaves the set; a promoted candidate joins — no restart.
3. **Hot path unchanged**: gRPC watch → signer-match → `intel.get(leader)` →
   size/exit/landing. One O(1) lookup, Rule-0 clean (the LaneDecision pattern).
4. **Fail-safe**: file missing/garbled ⇒ keep last good map; flag off ⇒ use
   `./wallets` + env exactly as today.

## 8. Feedback loop (self-sharpening when CT runs)

When trading, CT writes each outcome to the `fills` table (via the existing
`csv_writers` + `landwatch`): which leader, our Δslot, our CU, sell_reason,
realized ROI, and the competitor landings it observed. The scorer reads `fills`
as a **fast signal** to nudge tiers (did copying leader X actually pay *us*?),
exits (did our realized give-back match?), and tip-bids (are we landing ahead?).
Off ⇒ Dune-free indexer keeps everything current; On ⇒ it also self-tunes.

## 9. Automation / schedule

- `ct-indexer`: always-on PM2 process (one host, or the dev box).
- `scorer`: cron, nightly (ClickHouse data is seconds-fresh; nightly re-rank is
  enough; intraday optional). Emits `dashboard.html` + `leader-intelligence.json`.
- Distribution: rsync `leader-intelligence.json` to the 5 hosts (copy Snipping's
  `sync-tier-snapshot.sh` + crontab — proven pattern).

## 10a. BUILD STATUS (2026-05-30) — service is built

| Piece | Status | Artifact |
|---|---|---|
| ct-indexer | **BUILT, cargo check PASS** (measurement-only, never dispatches) | `src/bin/ct-indexer.rs` |
| ClickHouse schema + views | **BUILT** (trades + E9/positions/exits/landings views) | `observability/intel/clickhouse/schema.sql` + `docker-compose.yml` |
| scorer | **BUILT + runs** (cold-start verified: 194 leaders; E9 gate applied) | `observability/intel/scorer.py` |
| E9 creator-gate | **RAN** on 64 candidates (Dune, 0.62cr): 4 insiders→BENCH, 2→PROBATION | `research/dune/dash_data/q-cand-e9.json` |
| publish → Grafana | **BUILT + runs** (Loki + Prometheus + dashboard) | `observability/intel/publish_intel.py`, §12 |
| PM2 + nightly cron | **BUILT** | `ecosystem.intel.config.js`, `run-nightly.sh` |
| `leader_intel.rs` bot reader (§7) | **BUILT, cargo check PASS** — gated, hot-reload; wallet-set + exit overlay + per-leader CU wired at all 4 buy sites (PF+PS, inner+outer); byte-identical when off | `src/leader_intel.rs` + `MonitorCtx.leader_exit/leader_landing` |
| **Live ClickHouse loop proven** | ClickHouse UP (docker) → schema+views applied → scorer read it → E9 BENCHed genesis-slot insiders → published. ct-indexer release binary built + smoke-tested | 2026-05-30 |
| exit-recon giveback port (CH) | **NOT built** — cold-start Dune TSL used meanwhile | — |
| gRPC live tap | code-complete; needs real creds + IP-allowlisted host (server, not dev box) | — |

The whole `ClickHouse → scorer → leader-intelligence.json → publish → Grafana`
loop runs live; the bot consumes it via gated `leader_intel.rs`. Only the
indexer's gRPC→ClickHouse tap is infra-gated (creds + Corvus allowlist on a
server). Cold-start path still runs with zero infra.

## 10. Build sequence

1. **ClickHouse schema** (§2) — ½ day.
2. **ct-indexer** (§1) — fork CT's ingest/decode into a measurement-only binary — 2–3 days.
3. **Port the query suite** Dune→ClickHouse + wrap `build_dashboard.py` into `scorer.py` that also emits the json — 2 days.
4. **cron + rsync** (copy Snipping pattern) — ½ day.
5. **`leader_intel.rs` reader** (§7), gated, hot-reload — 1–2 days.
6. **Feedback** wiring (`fills` ingest) — 1 day.

**Cold start = this session's Dune snapshot** (already captured) seeds the first
`leader-intelligence.json`; the indexer takes over the ongoing refresh, so there's
no data gap while it accumulates its 30-day window.

## 11. Safety properties (why this is low-risk)

- **No money path in the brain** — `ct-intel` has no wallet/keys/send; worst case
  it writes a stale or bad config; it can never fire a trade.
- **Additive + gated** — `LEADER_INTEL_ENABLED=false` ⇒ CT is byte-identical to today.
- **Fail-safe reads** — bad/missing file ⇒ last-good map; the bot keeps trading.
- **Decoupled** — brain down ⇒ bot runs on the last file; bot down ⇒ brain keeps
  collecting (Dune-free, off the always-on gRPC feed).

---

## 12. Grafana storage & rendering (the human view)

> **Decision (2026-05-30):** the intelligence is stored and rendered in the
> *existing* CT observability stack — not a standalone HTML file. ClickHouse
> stays **scorer-internal** (the raw firehose + heavy analytics that *produce*
> the intelligence); the **derived** result is published into Grafana's own
> services. Implemented in `observability/intel/` + one dashboard JSON.

**Why the split is forced.** `observability/docs/metrics-contract.md` forbids
full base58 addresses as Prometheus labels. So:

| Grafana store | Holds | Cardinality |
|---|---|---|
| **Loki** (`{job=ct-intel}`) | full per-leader registry — base58 + every field in the JSON line **body** | low-card labels only |
| **Prometheus** (`ct_intel_*`) | headline gauges + per-leader trends for the **active copied set** (tier ≥ SECONDARY), 8-char short code, capped 24 | bounded |
| ClickHouse/DuckDB (§2) | raw firehose + analytics — **not** a Grafana datasource | scorer-internal |

**Pipeline** (additive to §0):

```
leader-intelligence.json ─► publish_intel.py ─┬─► ct-intel.jsonl ─(promtail)─► Loki ─┐
                                              └─► ct-intel.prom  ─(exporter)─► Prom ─┴─► Grafana "CT bot" folder
```

- `observability/intel/publish_intel.py` — converts §6 json → Loki JSONL +
  Prometheus textfile. `--from-dashdata` cold-starts off this session's Dune
  snapshot (runs **today**; verified: 194 leaders, 104/69/13/8 by tier).
- `observability/intel/intel_exporter.py` — passive sidecar serving the textfile
  on `:9131/metrics` (same shape as `observability/exporter/exporter.py`).
- `observability/stack/grafana/dashboards/ct-leader-intelligence.json` — pushed
  into the **"CT bot"** Grafana folder by the existing
  `scripts/push-ct-dashboards.sh`. Panels: cohort/tier stats, the "magic
  recipe" active-set table (Prometheus), tier-drift trends, and the full
  base58-searchable registry (Loki).
- Wiring snippets: `ct-intel-jobs.yml` (Prometheus scrape) +
  `intel/promtail-intel.snippet.yml` (Loki tail). New series registered in
  `metrics-contract.md` under "Intelligence service (`ct_intel_*`)".

This **supersedes** the standalone `dashboard.html` as the durable human view
(the HTML stays as the offline research artifact). The bot path is unchanged —
it still reads `leader-intelligence.json` (§7); Grafana is purely the human lens
on the same file.