Design — Message Management System

Overview

CONTEXT.md states what this system does. The decision records in docs/adr/ state the architecture. This document states how to build it.

The architecture decisions stay closed here. This design applies them. It also settles the questions the decision records left open, and it records each answer beside the part of the system that answer governs.

Key Decisions

Choices the decision records left open

Choices settled in the review of 2026-08-14

Toolchain prerequisites

"experimentalDecorators": true,   // legacy decorators — the shape Nest is written against
"emitDecoratorMetadata": true,    // design:paramtypes — Nest DI reads this

The probe confirms two results when both flags are present. Decorators run, and the transpiler emits design:paramtypes. ESM, verbatimModuleSyntax and allowImportingTsExtensions all work beside them. Parameter types still resolve across a .ts import, so the repository keeps its current compiler settings.

TypeScript stays at ^5.9. Version ^7 is the native tsgo compiler. Both ts-jest and typescript-eslint target the 5.x compiler API. nub transpiles at run time, so TypeScript here serves only the type check, the lint and ts-jest. Version 7 adds nothing, and it can break three tools at once inside the timebox.

Write the skeleton by hand. nest new selects its own package manager and writes npm-flavoured scripts, which ADR-0003 forbids. It also generates the flat feature shape that ADR-0005 calls a trap. Slice 0 runs nub add for the Nest packages instead, then creates main.ts and app.module.ts directly in the four layers.

Where the tenant context starts

ADR-0012 first placed this work in middleware. Middleware runs before the guard, so the token is still unverified at that point. The guard cannot do the work either, because canActivate returns a boolean and the store closes as soon as it returns.

What the read ports return

Message.create generates the id and the timestamp, so it cannot rebuild a stored message. A read path therefore needs a second construction route.

When sanitization runs

ValidationPipe calls plainToInstance before it validates. A @Transform therefore runs ahead of every validator. That order causes two faults.

• The bounds would measure the sanitized text. sanitize-html escapes text entities, so & becomes & and < becomes <. A valid 4000-character message of ampersands grows to about 20 000 characters. The API then rejects a length the client never sent.
• The blank check in Message.create would never fire over HTTP. The DTO always catches the case first, so the entity invariant becomes decorative.

Build slices

Build one vertical slice at a time, endpoint by endpoint. End every slice with nub run test green, nub run lint green, and exactly one commit.

Slice 0 also carries docker-compose.yml and a connectivity smoke test. The riskiest infrastructure surprises then appear early, although the code that uses Kafka and Elasticsearch arrives in slice 4.

Architecture

Module and layer map

src/
  main.ts                    ValidationPipe(whitelist, forbidNonWhitelisted, transform),
                             exception filter, Swagger, enableShutdownHooks, listen
  app.module.ts              module wiring + APP_GUARD (global JwtAuthGuard)

  domain/                    imports nothing from the other three layers
    message.ts               entity — private ctor, static create(), immutable
                             WRITE PATH ONLY (ADR-0020)
    message-view.ts          read projection — plain immutable type, no behaviour
    message-created.event.ts event payload
    cursor.ts                (timestamp, id, direction) — encode/decode, rejects malformed
    page.ts                  Page<T> { items, nextCursor }
    errors.ts                DomainError + typed subclasses
    uuid-v7.ts               v7 generator (node:crypto randomBytes; NOT randomUUID)
    ports/
      message-writer.port.ts       save(message: Message)
      message-reader.port.ts       listByConversation(...) : Page<MessageView>
      message-searcher.port.ts     search(...) : MessageView[]
      message-indexer.port.ts      index(message)
      event-publisher.port.ts      publish(event)
      identity-context.port.ts     require() : { tenantId, senderId }

  application/               @Injectable() lives here and nowhere below
    create-message.usecase.ts
    list-conversation-messages.usecase.ts
    search-conversation-messages.usecase.ts
    index-message.usecase.ts          driven by the consumer, not by HTTP

  infrastructure/            implements the ports
    config/         env.config.ts (class-validator), config.module.ts
    identity/       als-identity-context.ts        AsyncLocalStorage adapter
                    identity.interceptor.ts        APP_INTERCEPTOR — enters the store
    mongo/          client provider, message.mapper.ts (_id <-> id, the ONLY place),
                    mongo-message.repository.ts (writer + reader), create-indexes.ts
    kafka/          kafka.client.ts, kafka-event-publisher.ts,
                    message-created.consumer.ts (onModuleInit / onModuleDestroy)
    elasticsearch/  client, message-index.mapping.ts, es-message-index.ts (indexer + searcher)
    auth/           jwt.strategy.ts, jwt-auth.guard.ts, public.decorator.ts

  interfaces/http/
    messages.controller.ts               POST /api/messages
    conversation-messages.controller.ts  GET list + GET search
    health.controller.ts                 the only @Public() route
    dto/                                 create, list-query, search-query, message-response
    sanitize-content.pipe.ts             runs after ValidationPipe
    domain-exception.filter.ts           DomainError -> HTTP status

The top level orders by layer, per ADR-0005. Slice 0 configures the ESLint import/no-restricted-paths zones from that decision record. Those zones must pass before every commit.

Six single-method ports, not one repository

This takes interface segregation literally, per ADR-0006. IndexMessage receives MessageIndexer, so it cannot reach save at all. MongoMessageRepository implements the writer and the reader in one class, because both share the collection handle and the mapper. No other implementation is shared.

IdentityContext

ADR-0012 shows tenantContext.require(). The create use case also needs senderId, and ADR-0018 forbids the controller to pass identity into the use case. The port therefore carries both claims and takes the name IdentityContext.

// domain/ports/identity-context.port.ts
export interface IdentityContext {
  require(): { tenantId: string; senderId: string };   // throws when absent
}

One adapter implements the port. The use cases and the Mongo repository both use that adapter. The rule does not change: an absent context throws, and it is never a wildcard. ADR-0012 carries a dated amendment for the new name.

The interceptor enters the store

A global APP_INTERCEPTOR is the only place that runs after verification and also wraps the call. It reads request.user, which the passport strategy sets, and it wraps next.handle().

// infrastructure/identity/identity.interceptor.ts
intercept(ctx: ExecutionContext, next: CallHandler) {
  const { user } = ctx.switchToHttp().getRequest();
  return new Observable((sub) =>
    als.run({ tenantId: user.tenantId, senderId: user.senderId },
      () => next.handle().subscribe(sub)));
}

The @Public() health route carries no user, so the interceptor enters no store. Any later call to require() then throws. That is the intended fail-closed behaviour.

The event pipeline

• The application creates the topic at startup through the kafkajs admin client. Kafka auto-creation yields a single partition, which silently destroys the partitioning design. The application asserts the topology instead of inheriting a broker default.
• The producer sets idempotent: true and acks: all. That one flag removes duplicates from producer retries. The consumer-side rule in ADR-0011 still applies, because it covers redelivery and not only retries.
• Offsets commit only after the index call resolves. kafkajs commits after eachMessage returns. A throw means no commit and a redelivery, and the upsert by id makes that redelivery harmless. No manual commit code is needed. Never catch an error and continue inside the handler, because that commits past a failure.
• The consumer reads the tenant from the payload, never from the store.
• The consumer starts on onModuleInit and stops on onModuleDestroy. enableShutdownHooks() lets SIGTERM leave the consumer group cleanly, instead of a wait for the session timeout.
• The topic name, the partition count and the group id live in configuration, not in string literals (ADR-0010).

The system must make that failure visible. One process holds both the API and the indexer. A halted consumer would otherwise leave the API at 201 while search quietly stops. The consumer records its own liveness, and the health route reports it.

GET /health -> 200 { status: 'ok',       indexer: 'running' }
            -> 503 { status: 'degraded', indexer: 'stopped',
                     reason: 'consumer crashed' }

this.consumer.on('consumer.crash', (e) => {
  this.state = 'stopped';
  this.logger.error('indexer halted', e);
});

This check is hand-written. @nestjs/terminus is an official Nest package, but it would be a dependency for one boolean. The check also gives the @Public() health route a purpose beyond "the process is up".

One process, not two

The consumer starts inside the API process on bootstrap. One command runs the system, and one command demonstrates it. Integration tests then exercise the whole pipeline in a single Nest testing module. The cost is that the API and the indexer cannot scale apart, and the README names that cost. The consumer group already exists, so a split entry point is later a bootstrap change and not a redesign.

Data Model

Field bounds

Oversized content is a required edge case under ADR-0014, so the limit is a stated number and not an implicit one. The same reason gives metadata a number of its own. MongoDB stores metadata verbatim, and the Elasticsearch _source keeps it too, because enabled: false indexes nothing but stores everything. An unbounded object therefore costs twice.

The type stays Record<string, any>. CONTEXT.md declares that type verbatim in the data model. A narrower type of primitives only would depart from the given contract rather than harden it.

@IsOptional() @IsObject() @MaxJsonBytes(4096)
metadata?: Record<string, any>;

Two checks on content, and both fire

sanitize-html can reduce <script>alert(1)</script> to an empty string. Sanitization therefore creates the blank case. It does not only pass one through. The two checks catch different inputs, and the pipe order makes each one reachable.

The second check is the one genuine invariant of the entity under ADR-0016. It is not redundancy. Both checks get a test, and the integration suite asserts that the script case returns 400 rather than an empty stored message.

Identity and the clock

The domain generates id before persistence, per ADR-0008. The same id then keys both stores. timestamp is server-assigned and is not a request field.

One clock read serves two uses. Message.create reads Date.now() once. It derives the v7 id and the timestamp from that single value, so the millisecond inside the id always matches the exposed timestamp. ADR-0013 rests on exactly that agreement when it calls the tiebreaker meaningful.

static create(props: CreateProps): Message {
  const now = Date.now();
  return new Message(uuidV7(now), new Date(now), /* … */);
}

There is no Clock port. Such a port protects no invariant, which ADR-0016 forbids. Tests get determinism from Jest fake timers.

The MongoDB index and the cursor

• Read limit + 1 rows, and run no count query. The extra row shows whether a next page exists, and it supplies the cursor for that page. A countDocuments beside every page would scan the whole conversation to answer what one extra row answers for free.
• The cursor encodes its own direction. ADR-0013 records that a sort change invalidates an outstanding cursor. The cursor therefore carries enough to detect the change. A desc cursor with sort=asc returns 400, not a wrong page.
• The mapper converts the id in the cursor, not the repository. The keyset bound needs a BSON UUID built from the canonical string in the cursor. That conversion in the repository would be a second translation site and would break the single-place rule of ADR-0008. The mapper exports toBinaryId and toStringId, and the repository calls them. The repository still names _id in the filter and the sort, which is unavoidable, but it never knows the representation.
• The tiebreaker works because Binary sorts the way v7 sorts. MongoDB orders BinData by length, then by subtype, then byte by byte. Every message _id is 16 bytes of subtype 4, so the comparison is byte-wise. A UUIDv7 leads with a 48-bit timestamp in big-endian order, so byte order is time order. That order matches the canonical string order the cursor carries. This fact is load-bearing, so it gets a test that inserts ids generated in the same millisecond and asserts a stable total order.
• One index serves both directions: (tenantId: 1, conversationId: 1, timestamp: -1, _id: -1). MongoDB walks an index backwards at the same cost, so asc needs no second index. _id sits fourth, so the tiebreaker sorts inside the index and not in memory.

The Elasticsearch mapping

{
  "dynamic": "strict",
  "properties": {
    "tenantId":       { "type": "keyword" },
    "conversationId": { "type": "keyword" },
    "senderId":       { "type": "keyword" },
    "timestamp":      { "type": "date" },
    "content":        { "type": "text", "analyzer": "standard" },
    "metadata":       { "type": "object", "enabled": false }
  }
}

hits.map((h) => ({ id: h._id, ...h._source }))   // -> MessageView (ADR-0020)

dynamic: strict turns an unexpected top-level field into a hard error, instead of a quiet addition to the mapping. That is the same failure that enabled: false prevents for metadata, now closed at the root as well. Strictness does not reach inside metadata, because enabled: false means Elasticsearch never parses the content.

metadata is Record<string, any> with client-controlled keys. Dynamic mapping would let one tenant grow the cluster-state mapping with varied keys, until the index becomes unusable. enabled: false keeps the object in _source, so results still carry it, and indexes none of it. This is the concrete reason the spec asks for explicit mappings, and it belongs in the README.

The analyzer is standard, not english. Stemming helps English recall and quietly damages every other language, and this corpus is general chat.

One index holds all tenants, and a filter scopes each query. There is no index per tenant. That trade-off belongs to ADR-0012. The application creates the index with the mapping at startup if the index is absent.

API Design

The create path

The guard verifies ES256 with the algorithm pinned, and it checks iss, aud and exp. It rejects a token that lacks tid or sub. The DTO accepts conversationId, content and an optional metadata. A body that carries senderId returns 400. The API never ignores that field in silence.

201 { id, conversationId, senderId, content, timestamp, metadata }

The pipes register in this order, and the order is the point.

// main.ts
app.use(json({ limit: '256kb' }));          // sanitize-html never parses a megabyte
app.useGlobalPipes(new ValidationPipe({ whitelist: true,
  forbidNonWhitelisted: true, transform: true }));
app.useGlobalPipes(new SanitizeContentPipe());

A failed publish still returns 201

The message sits in the system of record, so the request succeeded. A failure response would invite a client retry, and that retry writes a second message under a new id. The service logs the failure at error level and never swallows it, per ADR-0011. The cost is that the message stays absent from search until a backfill. ADR-0011 already accepts that gap, and the README names it beside the note on the transactional outbox.

kafka.producer({
  idempotent: true,                                 // implies acks:-1, maxInFlight:1
  retry: { retries: 2, initialRetryTime: 100 },
});
await producer.send({ topic, timeout: 2000, messages: [...] });

The API still waits for the send, so the save precedes the publish. The unit test that asserts that order stays meaningful.

List a conversation

GET /api/conversations/:conversationId/messages?limit=&sort=&cursor=

limit   default 20, 1..100           sort  asc | desc (default desc)
cursor  base64url of (timestamp, id, direction) — opaque, parsed defensively

Mongo:  { tenantId, conversationId, $or: [ keyset ] }
        .sort({ timestamp: dir, _id: dir }).limit(limit + 1)
-> 200  { items: [...], nextCursor: string | null }

Search a conversation

GET /api/conversations/:conversationId/messages/search?q=&limit=

ES:  bool {
       must:   [ { match: { content: q } } ],        scored
       filter: [ { term: { tenantId } },             filter context — unscored, cacheable
                 { term: { conversationId } } ]
     }
-> 200  { items: [...] }        relevance order

Both read endpoints return an envelope. Two endpoints over one resource must not return different container shapes. The envelope also leaves room for search_after later as an additive change instead of a breaking one. An empty result is 200 { items: [] }, and that case is a required edge case.

The query uses match against a DSL object. Never use query_string, and never concatenate strings, per ADR-0015. The API validates q as non-blank and 256 characters or fewer.

The endpoint returns the hit directly and does not hydrate from MongoDB, so search costs one round trip. Elasticsearch stays a derived read model under ADR-0009. The system reads it for search and nothing else, and MongoDB can rebuild it. The trade-off is that a lagging index serves slightly stale content. That is acceptable for a search result, and the README names it.

Search has no pagination. It accepts limit only, capped at 100. The spec asks for pagination on the listing endpoint, not on search. Deep relevance paging needs search_after, and the README names that instead of a half-built version.

Domain errors map to two statuses

Input faults return the same 400 the DTO returns. A client then sees one status for one class of problem, whichever layer caught it. An absent identity context is deliberately not a 401. The guard already ran, so this case means a bug in our wiring. A 401 would send the caller to check a token that was correct.

A cross-tenant read is empty, not 404

Tenant scope is part of the query under ADR-0012. Another tenant's conversation therefore does not exist from this caller's view.

GET /api/conversations/abc/messages     token: tenant B, conversation abc: tenant A
200 { "items": [], "nextCursor": null }  — identical to a conversationId that never existed

Nothing is special-cased and nothing leaks. A 404 would need an existence check that the design has no other reason to run. It would also confirm that some tenant owns that conversationId. That is a cross-tenant existence oracle, and tenant isolation exists to withhold exactly that.

Testing Strategy

Follow TDD throughout, per ADR-0014. Write the failing test first. Confirm that it fails. Then write the minimum code that makes it pass.

Unit tests — no infrastructure, every port faked

• Message.create invariants — blank content, and content that the sanitize pipe reduced to nothing before it arrived
• Message.create clock coherence — the millisecond inside the id equals timestamp, with Jest fake timers
• Cursor round trip, malformed input, direction mismatch
• CreateMessage — the save precedes the publish, and a rejected publish still returns the message
• ListMessages — the extra row becomes nextCursor and never appears as an item
• SearchMessages
• IndexMessage

Every fake honours the same contract as the real adapter. It throws the same errors, keeps the same order, and adds no stricter precondition. ADR-0006 requires this under the Liskov rule.

Integration tests — compose stack, Nest testing module, supertest

Assert the query plan

An integration test checks that the winning plan is an IXSCAN with no SORT stage. A later change to the index or the sort then fails the build instead of a quiet slowdown. A nub run db:explain script prints the same plan for the README.

const plan = await collection.find(keysetFilter)
  .sort({ timestamp: -1, _id: -1 }).explain('queryPlanner');
expect(stages(plan)).toContain('IXSCAN');
expect(stages(plan)).not.toContain('SORT');

Three mechanics to fix now

1. Jest global setup generates the test keypair. Never commit it, not even as a fixture (ADR-0018).
2. Each integration run uses a fresh random tenantId. About three lines then remove the limitation in ADR-0014 about compose state that persists between runs.
3. Integration tests run one at a time, in their own Jest project, under a random id for the consumer group. Jest runs test files in parallel workers by default, and slice 4 puts a Kafka consumer inside the app. Every integration file would then start another consumer in the same group, which rebalances mid-test and moves partitions between workers. maxWorkers: 1 keeps one consumer alive at a time. The random group id keeps the committed offsets of a previous run out of this one. Unit tests stay parallel and free of infrastructure.

// jest.config
projects: [
  { displayName: 'unit',        testMatch: ['**/*.spec.ts'] },
  { displayName: 'integration', testMatch: ['**/*.int-spec.ts'],
    maxWorkers: 1, globalSetup: './test/setup.ts' },   // keypair + run ids
]

One command still runs both projects: nub run test.

Out of Scope (MVP)

ADR-0017 sets this list, and this design confirms it. The build includes none of the following.

• Caching
• Rate limiting
• A refresh-token flow
• A users collection
• Roles
• A dead-letter queue
• A transactional outbox
• Deep paging with search_after

The README names each item with its reason. Where ADR-0017 ranks an item, the README states the rank.