AI on your computer

Set up AI: the five paths

The Set up AI wizard opens from the Set up AI… button under Settings → AI Assistants, from the empty chat's Connect an AI… and when you use Ctrl+K with no AI. The first page asks what you want to set up:

• AI in the cloud — OpenAI, Gemini, Anthropic and others, with an API key. The simplest path and the best quality, and still the recommended default. The page is the same as the connector's: the connection is tested before it is accepted and the key is stored encoded only on this machine.
• AI on your computer — with Ollama or LM Studio — for those who already use (or want to use) one of these programs. The wizard says whether the program is installed, stopped or running, offers to install it, lists the models already installed and the ones recommended for download, with the download progress, and runs a quick test.
• AI on your computer — directly through LightNote — LightNote downloads the official llama.cpp program and a model, and runs both without you installing anything else. Before downloading, it shows how much disk and memory the model takes and measures the speed on your computer.
• Command-line assistant — Claude Code, Codex, Gemini CLI and others, to use in the terminal. In the chat and in Ctrl+K they answer all at once, without streaming and without LightNote's tools.
• Search by meaning — the embeddings that let the AI Chat find notes by the meaning of the question, computed in the cloud or on your computer.

Every path ends on the Done page, with the AI's name (the name it appears under in the chat, in Ctrl+K and in Settings), the list of models and the Use as Default AI option. Opened from Settings, the wizard only saves when you click OK; opened from anywhere else, it saves when you finish. If a row for the same provider already exists without a key, it completes that row instead of creating another one next to it.

Model types, GGUF and quantization

Two types of model. A chat model (instruction-tuned, "instruct") writes: it is what answers in the chat, in Ctrl+K and in Block AI. An embedding model writes nothing: it turns each chunk of text into a vector of numbers, and texts with similar meanings get nearby vectors — that is what powers search by meaning. One does not replace the other, and the wizard only offers models of the right type on each path.

GGUF. It is the single-file format that llama.cpp, Ollama and LM Studio read: the model's weights and its description in one .gguf file. Quantization means storing each weight with fewer bits so the file is smaller and faster. The name says how many: Q8 uses 8 bits per weight and stays very close to the original; Q4 uses about 4, takes almost half the space and loses little. In our search tests in Portuguese, the Q4 and Q8 of the same model tied, and Q4 was nearly twice as fast. That is why the catalog prefers Q4 (Q4_0 or Q4_K_M).

Disk, memory, speed and graphics card

Disk is the size of the downloaded file, taken up for as long as you keep it. Memory is what the model uses while it is loaded: the weights plus room for the conversation, a little above the file size. LightNote runs the model in a separate process that only exists while you use the AI: after 5 minutes without use it shuts down by itself and the system gets all the memory back. The first question after that takes a bit longer, because the model is loaded again — the chat shows "Loading the model…". Speed depends on the processor and the size of the model: a smaller model answers faster, a larger one writes better.

Processor or graphics card. By default the model runs on the processor, which works on any computer (with AVX2 it is much faster). On the hardware page, Use the graphics card (Vulkan) downloads the version of the program that uses the graphics card, from any manufacturer. It may or may not be faster, depending on the card: the performance test at the end shows the real speed, and switching later is just a matter of going through the path again.

Where to find models and how to use another one

The simplest way is the LightNote catalog, on the Choose a model page: each of its files was checked by us, with size and sha256, and the list shows disk, memory, license and the badge of each one. The ones that fit well on this computer appear first. Outside it, models live on Hugging Face (huggingface.co, search for "GGUF"), in the Ollama library (ollama.com/library) and in LM Studio's own catalog; for the last two, use the Ollama or LM Studio path.

Using another model. In Other model…, paste the address of a .gguf file on Hugging Face (in the format https://huggingface.co/owner/repository/resolve/main/file.gguf) or choose a .gguf that is already on your computer. LightNote asks Hugging Face for the file's size and sha256 (without the sha256 it refuses, because it would have no way to verify the download) and reads the model header to estimate the memory. Before using it, check three things on the model page: the size (a file larger than your free memory won't run), the license and the type — for chatting, an instruction-tuned model ("instruct", "chat", "it"); for search by meaning, an embedding model ("embedding"). Prefer files published by the model's own author: a botched conversion may even run and return meaningless results.

Licenses

Each model has its own license, chosen by the author: Apache 2.0 and MIT allow free use; the Gemma Terms of Use (for EmbeddingGemma) and the Llama 3.2 Community License restrict some uses and require a notice. That is why the Terms of use page shows the license of the model and that of the program that runs it (llama.cpp, MIT), with a link to the full text, and the required notice, when there is one, in the original language. To continue, check I have read and accept the terms. LightNote does not redistribute any model: it downloads the file directly from the source (Hugging Face, and the program from llama.cpp's GitHub) and verifies the sha256 of each one before using it. An interrupted download resumes where it stopped.

Performance test and the badge

The badge. In the list of models, Fits well means the model uses less than half of the memory free right now; May be slow, that it fits but is tight (more than half of the free memory, or a processor without AVX2); Doesn't fit, that it would need more than 80% of the free memory — the wizard won't let you choose that one. Free memory changes with what is open: close heavy programs and the badge may improve.

The performance test runs the model once on this computer, so you can see what it costs before adopting it, and says everything in plain language: how much disk it takes, how much memory it uses while loaded, how many words it writes per second and how much of the processor it used. For an embedding model, it measures how many chunks it embeds per second and how long indexing the open folder would take. The sentence at the top sums it up: Good for chatting, Usable, but slow or Slow on this computer. The first measurement includes the time to load the model.

Search by meaning in the cloud or on the computer

On the Search by meaning path, the wizard asks where to compute. In the cloud (recommended): fast; only connected AIs that compute embeddings appear, and if none does, it tells you which one to connect. The provider charges per chunk on the first indexing and, after that, only for what changes; the wizard shows how many chunks the folder has. On your computer: free and private, but slower, and it uses memory while indexing — directly through LightNote (with the catalog's embedding models, EmbeddingGemma first) or through Ollama. The test says how long the first indexing of the folder would take.

Indexing on the computer runs in the background, with 2 cores and low priority, so as not to compete with you for the processor; later runs only redo what changed. Switching models reindexes the folder: vectors from different models can't be compared. The old index is kept, so going back to the previous model costs nothing. Settings shows a summary (model, where it runs and how many chunks are indexed) with Change…, which reopens the wizard.

Model list and the remembered assistant

One list of models per row. Each registered AI is a row with a unique name, and the model field accepts a list: the first is the preferred one, and the following ones are tried, in order, if it fails. In the Settings table, separate them with commas; in the wizard and in the connector, use the move up, move down and remove buttons, and Fetch models… asks whether the chosen model joins the list or replaces it. Your list applies even with automatic model switching turned off (automatic switching only adds the catalog's alternatives after it). When the first one fails and another answers, a notification says which one failed, why and which one answered; clicking it offers to remove the model from the list — LightNote never changes your list on its own.

The remembered assistant. The chat, Ctrl+K, Block AI and the AI windows remember the last assistant you chose in each of them, per window and folder, and bring it back when you reopen the app — as long as it is still available. The Default AI does not change because of this: it only changes when you switch it in Settings. In the selection lists, the AIs are grouped by provider.

Comparing answers

In the chat, Compare answers… (or /compare before the question) sends the same question to 2 to 4 assistants at once, with the same folder context, and shows the answers side by side, each column saying whether the assistant is local or in the cloud. Under each one: the time to the first chunk, the total, tokens per second, tokens and, for a local model, the memory. Continue with this one puts only the chosen answer into the conversation and switches the chat to that assistant; the others are discarded.

In Ctrl+K, Compare with… sends the same instruction to 2 to 4 assistants and shows the proposals side by side, with the changes highlighted against the selection. Nothing is applied to the document before you choose; Apply this one applies it once, and Ctrl+Z undoes it. The comparison is read-only: no tools, no automatic model switching and no failure dialog — a failure in one becomes a sentence in its column. Remember that a cloud assistant receives the text sent.

Status bar, unloading and privacy

While a model run by LightNote is loaded, the status bar shows Local AI with the model's name and the memory it uses (or "loading…"). Click it to Unload now and give the memory back right away, without waiting the 5 minutes; the next question loads it again. The indicator disappears when no model is loaded.

Privacy. With a model on your computer — directly through LightNote, through Ollama or through LM Studio — nothing leaves the machine: not the question, not the notes, not the search vectors. The model's program listens only on the computer's local address, with a new password each time it is started, so that no page open in the browser can talk to it.