Skip to content
CompanyWizard

PII pseudonymization for embeddings (scrub before embed)

Admin

The PII shield protects your data on four egress paths: chat prompts, the retrieved knowledge context (RAG) and the results of reading tools are protected as soon as PII detection is switched on. The fourth path — the embedding calls made when indexing the knowledge base and when running vector search — is an opt-in for administrators. This page covers it.

When scrub before embed is active, detected personal values go to the embedding provider as type placeholders ([PERSON], [IP_ADDRESS], …) on the embedding path too — never in clear text.

Do I need this?

Section titled “Do I need this?”

The question is the same as for trusted targets: do the embedding data leave my area of responsibility?

  • Yes — an untrusted cloud embedding provider: otherwise the provider sees clear-text PII while indexing and searching. This is what the feature is for.
  • No — a local provider (e.g. Ollama, LM Studio): no egress takes place. You would typically mark such a provider as trusted — the feature is then deliberately inactive: even if you set the switch, it does not kick in while the embedding provider is trusted. → Keeping data local

Trusted is not local. The checkbox does not turn a cloud provider into a local one — it permits it to receive original text. Mark a cloud provider as trusted and clear text leaves, even with this switch on.

Prerequisites for enabling it

Section titled “Prerequisites for enabling it”
  1. PII detection (Presidio) set up and active. Without it nothing can be pseudonymized. → Set up Presidio
  2. The embedding provider is NOT marked as PII-trusted — neither via the “Trusted embedding provider” checkbox on its row under Models nor, if embeddings run via the configuration, via “Embedding configuration (Embeddings:*) is trusted” in the system settings. If it is trusted, the feature has no effect (see above). → Trusted targets
  3. Leave hybrid search on. It is the reason value searches keep working despite pseudonymization (see below).

You will find the “Scrub before embed” switch in the PII detection admin section.

What happens when you enable it

Section titled “What happens when you enable it”
  • A confirmation dialog with a duration hint — the next step is heavy.
  • An automatic full reindex of the knowledge base. Everything is re-embedded, this time pseudonymized. On large knowledge bases this can take hours; an x/y indicator in the top bar shows progress.
  • On the desktop the reindex runs only while the app is open. If the app is closed it pauses and resumes automatically after a restart (already-processed content is skipped).

This applies in both directions: a reindex runs both when enabling (off ⇒ on) and when disabling (on ⇒ off), because the embedded text changes either way.

For the same reason CompanyWizard rebuilds the vector index completely when you toggle the trust checkbox of the active embedding provider while scrub before embed is enabled. → Trusted targets: model providers

Section titled “Effect on search”

The key point — and the reason hybrid search is required:

  • Semantic search: unchanged. Type placeholders instead of real values cost no semantic recall (identical in measurements). Questions about meaning and context work just as well.
  • Searches for concrete values (IP addresses, names, email addresses) are carried by the local keyword search (BM25) instead of vector search. BM25 indexes content raw and locally — the real value is never hidden from it. So do not turn hybrid search off: without it, value search would fail.
  • Stored content, display and citations stay unchanged. Only what goes to the embedding API is replaced. What you read in the knowledge base and what gets cited remains the original text.

Limits — honestly stated

Section titled “Limits — honestly stated”
  • Risk reduction, not a guarantee. As everywhere, the protection relies on PII detection; it can miss unusual formats. → Limits of the PII shield
  • Fail-closed on PII-service outage. If Presidio fails, affected documents are not embedded, and vector search may return fewer or no hits for them — but no clear data ever reaches the provider. Keyword search keeps running, and an affected document is retried on the next run.
  • Imports under “PII Shield Off”. If a user promotes an import while they have switched the shield off, those documents land in the index without pseudonymization — and stay that way until their content changes or a full reindex runs. Automatic syncs are not affected. You lock the switch in the system settings. → The PII shield
Section titled “Related”