Financial services — security operationsRAG & zero-shot classification

Zero-shot document classification that gave SOC analysts their day back

Security analysts were hand-labelling documents in Confluence instead of doing security. Zero-shot classification meant no labelling bottleneck and no training-data wait.

An explanation of the central design choice. The supervised path is shown as define categories, hand-label thousands of documents, train, deploy — with a loop back showing that the labelling would have been done by the analysts whose time was the constraint in the first place. The zero-shot path shipped instead runs category descriptions written as prose, RAG over the client own documents as an open-book test rather than model recall, and a label returned with a confidence score such as Form 8-K at 95 percent, classified Regulatory Critical. Below: 85 percent classification automated, 3 weeks to MVP on 1.0 full-time equivalent, and zero net cost because AWS funding covered the engagement.
A supervised classifier would have started by creating more of the exact problem it was meant to solve.

Project snapshot

Client
A financial services company with an in-house security operations centre
Industry
Financial services
Business function
Security operations and document governance
Challenge
SOC analysts were manually classifying documents in Confluence, losing hours to search and applying inconsistent labels across thousands of files. Security experts were doing admin work instead of security, and document volume kept growing with no scalable answer.
Solution
A RAG system that gives the LLM an open-book test against the client’s own documents, with a knowledge graph mapping concept relationships. Classification is zero-shot — category descriptions are enough — so there is no data-labelling bottleneck. Semantic search lets analysts find by meaning across policy docs, email threads and regulatory filings.
Result
Analyst time on document admin fell from 60% of the working day to 24%. Document processing went from manual hours to about two minutes each, 85% of it automated, at 85% classification accuracy against labels that had been inconsistent. Delivered as a 3-week MVP on a 1.0 FTE team at zero net cost.

Key outcomes

60%
Of analyst time reclaimed
85%
Classification accuracy
2 min
To process one document
3 weeks
To MVP, on a 1.0 FTE team

The client

Security experts doing filing

The security operations centre had a documentation problem that had quietly become a staffing problem. Policy documents, email threads, regulatory filings — thousands of files in Confluence, classified by hand.

By the people hired to defend the company.

The challenge

The most expensive people doing the least skilled work

Manual classification in Confluence, performed by SOC analysts. Hours lost searching for documents that existed and could not be found. Inconsistent labels across thousands of files, so the taxonomy degraded as it grew.

And document volume kept rising with no scalable answer in sight — which meant the problem was not stable, it was compounding.

The framing that mattered to the client: analysts were spending the majority of their time on document admin instead of protecting the company.

Industry context — published research, not our measurements

64%
of SOC analysts spend over half their hours on manual work they believe is automatable
69%
of security operations centres still rely on manual reporting
71%
of SOC analysts report burnout, and 64% consider leaving within a year

This is not a local problem, which is why the platform was built to be reused. Sources: Tines, Voice of the SOC Analyst and the SANS 2025 SOC Survey. They describe the sector, not this client — the measured result for this engagement is the drop from 60% to 24%, measured below.

Constraints

What we had to design around

  • No labelsA supervised classifier would have needed thousands of hand-labelled examples — produced by the same analysts whose time was the problem.
  • Team1.0 full-time equivalent across the entire engagement.
  • WindowThree weeks to a working MVP, not a prototype.
  • BudgetZero. The engagement had to run inside AWS funding, which ruled out anything with a per-seat or per-token commitment attached.
  • LocationConfluence is where the documents live, and would stay there. No migration.

Taken together these ruled out most of the obvious architectures before design started. Three weeks, one person and no budget is not a context in which you train anything.

Our approach

Zero-shot, because the labelling bottleneck was the problem

A supervised classifier solves the classification problem by first creating a worse version of the same problem. Someone has to label the training set. In this engagement, that someone was the constraint.

Why the supervised path loops back on itself

1,000s
of documents that would need hand-labelling to build a training set
1.0 FTE
the entire engagement team — no capacity to absorb a labelling phase

Zero-shot classification needs only category descriptions written as prose. No labelling, no data-collection phase, no analyst time consumed before deployment — which is what made three weeks possible.

The solution

RAG-powered document intelligence

The system gives the LLM an open-book test against the client’s own documents rather than relying on what the model already knows.

  1. RAG system — Retrieval-augmented generation over the live document corpus — an open-book test against the client’s own documents rather than model recall.
  2. Knowledge graph — Concept relationship mapping across the corpus, so related documents connect rather than sitting as isolated keyword matches.
  3. Zero-shot classification — Category descriptions are sufficient. No labelling phase, no data-collection wait, immediate deployment.
  4. Confluence integration — The system works where the documents already are, through the Confluence API.
  5. AWS — Enterprise-grade infrastructure, funded — so net cost to the client was zero.

Classification returns a confidence alongside the label. Asked to classify an SEC filing, the system answers with the structural and regulatory-language evidence, the specific form type, a 95% confidence, and the category it assigned.

The same retrieval layer serving two jobs. A RAG and knowledge graph foundation over the document corpus maps concept relationships across policy documents, email threads and regulatory filings. Job one is classification: zero-shot against written category descriptions, returning a label and a confidence rather than a bare guess. Job two is semantic search: a query about wire transfer limits returns twelve documents across three source types, including documents that never use the phrase. The same foundation now carries compliance automation and knowledge discovery — it was never a single-purpose classifier.
The second job is the one the analysts asked for once they had the first. Keyword search could only ever return documents containing the words typed.
The platform architecture as layers. On the left, users reach a UI that fronts text and voice interfaces, notification, LLM ops, an LLM registry and an LLM gateway, with LLM providers held in a separate column on the right. Below those run the shared horizontal layers: API management, the AI orchestration platform, monitoring and logging, notification ops, and then the data platform and core systems, with MCP servers alongside. The point of the shape is that classification is one consumer of a general platform rather than a standalone tool.
Classification was one consumer of a general platform, which is why the same foundation later carried compliance automation and discovery.
The retrieval pipeline. Source documents — wiki pages, web content, a data lake and file uploads — land in file storage, from which profile and text data, images and PDFs are separated. Feature extraction and embedding extraction run over them into a knowledge base. Below, a data ingestion path feeds a triplet generation stage driven by a triplet generation prompt template, whose output is indexed back into the same knowledge base. The knowledge base is drawn as three stores: customer data, policies and conversation history.
Two paths into one knowledge base — direct embedding extraction, and prompt-driven triplet generation that gives the graph its relationships.
12 documentsreturned for one question about wire transfer limits — across policy docs, email threads and regulatory filings, including ones that never use the phrase

Responsible by design

A label with a number attached to it

Automated classification in a security context has an obvious failure mode: a document quietly filed under the wrong category is harder to find than one that was never filed at all, and nobody discovers it until they need it.

So every classification carries a confidence score rather than arriving as a bare assertion. An analyst can see which labels the system was sure about and which it was not, and spot-check accordingly — which is a different and much cheaper review task than classifying from scratch.

The knowledge graph does related work: because concepts are mapped across the corpus, a document that sits oddly relative to its neighbours is visible as an anomaly rather than disappearing into a category.

Results

The day the security team got back

  • Analyst time on document admin fell from 60% to 24% of the day — a 60% cut in the admin load, and the number the engagement was actually judged on.
  • 85% of document processing automated, at roughly two minutes per document against manual handling measured in hours.
  • 85% classification accuracy, replacing labels that had been inconsistent across thousands of files. Search moved from keyword-only to semantic in the same pass.
  • A 3-week MVP on a 1.0 FTE team, at zero net cost, with the platform built as a reusable foundation rather than a point solution.
A four-column table comparing each metric before and after the system, with the measured impact. Document processing: from manual hours to two minutes per document, 85% automated. SOC analyst time: from 60% of the day on admin to 24% on admin, a 60% time saving. Classification accuracy: from inconsistent to 85% accurate, giving consistent quality. Search capability: from keyword-only to semantic search, so documents can be found by meaning. Delivered in 3 weeks on a 1.0 full-time-equivalent team, at zero net cost to the client.
The second row is the one that mattered to the client — admin time more than halved, and it went back to security work.
60% → 24%the share of an analyst's day spent on document admin, before and after — the rest went back to the work they were hired for

Beyond the numbers

What else changed

The RAG platform built for classification now supports work nobody scoped at the start — compliance automation for regulatory document processing, knowledge discovery connecting insights across documents, and semantic retrieval across the corpus.

It became the foundation the next AI project starts from, rather than something to be rebuilt.

$0net cost, on AWS funding — which is also why a reusable platform was the right thing to build rather than the cheapest possible classifier

Worth being precise about the headline claim, because it is easy to state loosely. The 60% is not 60% of the analysts’ total time — it is the reduction in the time they were losing to document admin, which fell from 60% of the working day to 24%. That is the measurement, and it is the one we would defend in a room.

If your specialists are doing classification work by hand

The first question is whether you need a trained model at all. Often the labelling effort costs more than the problem, and zero-shot is enough.