Automating Slack FAQs

Slack-to-Notion FAQ pipeline that detects repeated questions using embedding similarity

Stack Python, FastAPI, OpenAI, Claude, n8n, Notion
Source GitHub

TL;DR

  • System that watches a Slack channel, clusters semantically similar questions using embeddings, and auto-drafts FAQ entries in Notion when a question is asked 3+ times
  • Auto-replies in Slack when a new question matches an existing FAQ article with high confidence
  • Two components: a Python clustering API (FastAPI + SQLite + cosine similarity) and an n8n workflow that handles Slack monitoring, filtering, clustering, drafting, and auto-reply

The problem

Every company I've worked at has the same problem: people ask questions in Slack, the answers disappear into threads, and whoever runs IT becomes a human search engine. I've wanted to fix this since Slack first showed up at work. Back then we didn't have the tools to programmatically understand that "how do I change my password" and "where do I reset my login" are the same question.

At Precision Nutrition I did it manually. Answer a question, write an article, link to it next time the question was asked. It worked, but the detection step was me noticing patterns because I was answering everything. That doesn't scale.

Now the tools exist. Instead of a human noticing "people keep asking about the VPN," the system recognizes that three different phrasings of the same question have come in and drafts a FAQ entry automatically. The human still reviews and publishes, but the system does the pattern recognition.

View the repo on GitHub →

How it works

The n8n workflow showing three stages: Filtering (Slack Trigger, regex check, LLM question filter), Answering (cluster check, auto-reply branch, LLM FAQ writer), and Document (output formatting, Notion HTTP request, mark drafted, Slack notification).
The full n8n workflow: Filtering → Answering (with auto-reply branch) → Document.

A message arrives in Slack. The n8n workflow runs it through two filters: a regex check for question-like patterns, then an LLM call to Claude that determines whether it's a genuine FAQ-worthy question (filtering out greetings, rhetorical questions, and time-sensitive stuff like "what time is standup").

If it passes both filters, the workflow sends the question to a Python API. The API generates a vector embedding using OpenAI's text-embedding-3-small model, then compares it against every stored question using cosine similarity. If the similarity score exceeds 0.70, the question joins an existing cluster. If not, it gets stored as a new standalone question.

When a cluster hits three questions, the workflow fires the second half of the pipeline: Claude drafts a FAQ entry from the question variations, a Code node parses the output into structured fields, and an HTTP request creates a page in a Notion database with the draft FAQ, a "Needs Review" status tag, and metadata about the cluster. The cluster gets marked as drafted, and the Notion URL and answer text are stored alongside it.

Closing the loop: if a new question comes in that matches a cluster that already has a drafted FAQ — same 0.70 similarity threshold, but the cluster is flagged as drafted — the workflow branches and auto-replies in Slack with the stored answer and a link to the Notion article. No human needed. The question gets answered instantly from the FAQ that was built from previous instances of the same question.

A Notion page titled 'How do I connect to the company VPN?' showing auto-generated metadata: cluster count of 3, Needs Review status, and a draft answer about installing Cisco AnyConnect.
An auto-drafted FAQ entry in Notion, created after three similar VPN questions were detected.

Threshold tuning

The similarity threshold was the core technical problem. Too high and paraphrased questions don't match. Too low and unrelated questions cluster together.

I added a /debug endpoint that returns raw similarity scores between any input and all stored questions, then tested systematically with a small test set:

Question compared to "How do I reset my password?"Score
"Where do I change my password?"0.7487
"I forgot my password, how do I get a new one?"0.7292
"Password reset instructions?"0.7095
"How do I update my email address?"0.5461
"What time is the team meeting tomorrow?"0.1017

In a small test sample of common company questions, paraphrased versions scored between 0.70 and 0.75 — and anything above 0.70 is a match. Higher scores mean a stronger match, so the effective range is 0.70 to 1.0. Scores from 0.55 to 0.70 non inclusive are similar topic, but different questions so they are ignored. Different-topic questions dropped below 0.55, and unrelated questions fall below 0.15.

Design decisions

Two-stage filtering over one. The regex filter is free and instant. It catches most non-questions before the LLM ever gets called. The LLM filter handles the remaining edge cases that regex can't catch: rhetorical questions, greetings, time-sensitive messages. Running Claude on every single Slack message would be wasteful. Running regex alone would let too much through.

Dedicated API over n8n-only. n8n has built-in Data Tables that could store questions and eliminate the Python service entirely. I evaluated this and chose the separate API because cosine similarity over 1536-dimension vectors is significantly faster in Python with numpy than in JavaScript loops, the API endpoints are independently testable, and the clustering logic is reusable by other clients. For a small-scale personal tool, the n8n-only approach is viable.

Claude over GPT for structured output. I tested multiple OpenAI models for the FAQ drafting step. GPT-4o-mini and GPT-3.5-turbo both produced inconsistent or empty outputs when asked to follow a specific Question/Answer format. Claude followed the format reliably on the first attempt. The question filter node also uses Claude for consistency.

Auto-reply only when confident. The system only replies in Slack when two conditions are met: the question matches a cluster at ≥ 0.70 similarity, and that cluster already has a drafted FAQ. This means auto-replies only happen for questions that have been asked at least three times before and have a reviewed answer. Off-topic or novel questions never trigger a reply, they just get stored quietly.

Incremental clustering over batch. Questions get assigned to clusters in real time as they arrive. This is simpler and gives immediate feedback, but it means the system can't merge clusters after the fact if two separate clusters turn out to be about the same topic. A batch approach would handle that, but that adds complexity that isn't justified at this scale.

What this demonstrates

This project connects the same instinct behind the knowledge base I built manually — noticing that the same questions keep coming up and turning them into documentation — and makes it systematic. The detection and drafting is automated, the human stays in the loop for review and publishing, and once a FAQ is published, the system auto-replies to future instances of the same question. The full loop: detect repeated questions → draft a FAQ → review and publish → automatically answer the next person who asks. The technical parts (embeddings, cosine similarity, threshold tuning) serve an operational goal of reducing the time between "this keeps getting asked" and "there's a FAQ for that."


This project builds on the manual approach documented in the knowledge base case study.