Turning on the AI assistant on this site downloads about 700 MB onto your device and wants roughly 1.1 GB of your graphics memory.
That's a great deal to borrow from somebody who arrived wanting to know whether I take interim work.
So it asks first.
That is not how most of these work. You type into a small box on one website and the useful part happens somewhere else entirely. Your question, whatever context the site attached to it and the answer that comes back all travel through an inference service belonging to a third party. There is often nothing wrong with that arrangement. There is something wrong with it being invisible.
It works before it borrows anything
Open Ask about Milo and it loads a small search index built from what is already published here.
Nothing resembling a language model has started at that point. No weights are downloaded. Your question hasn't been sent to me or to anyone selling inference by the token.
It's ordinary browser-side retrieval. It finds the relevant parts of the site and hands you the links. For a decent share of what people actually want to know, that's the whole job.
What kind of fractional CTO work do I take on. Do I only work in events. What evidence is there behind the AI and automation claim. Am I open to something interim.
The non-generative version is deliberately useful on its own, because AI shouldn't be a tax you pay before a search box is allowed to work.
The model only arrives if you ask for it
Choose Enable local AI and the browser downloads an open-weight model and runs it on your machine using WebLLM. Currently Llama 3.2 1B Instruct, prepared for in-browser use.
Small, by the standards of anything you've read about this year. Still 700 MB and still over a gigabyte of graphics memory.
I picture you, the person it is being asked of. Someone on a train, on a phone, on the mobile network, deciding in about eleven seconds whether this website is worth their remaining battery. A personal site shouldn't help itself to that quietly.
Once it's on, the same browser-side search pulls the relevant extracts and hands them to the local model with your question. The model drafts a short answer. Question and answer stay on the device. There's no commercial API behind the drawer.
The model files still have to come from somewhere though. The hosts serving them see the ordinary technical details of any download, your IP address and your browser among them. Privacy-first ought to mean describing the boundary accurately rather than implying the internet stopped existing.
The model isn't asked to remember me
The fashionable term is retrieval-augmented generation. RAG, if you would like to sound current in a meeting.
The useful part is far less grand than the acronym. The model isn't asked to recall my career, work out my services or improvise a client list. It is handed the most relevant pieces of the published site and told to stay inside them.
Every answer keeps its sources visible.
Which matters, because a small local model can be confidently wrong. So can a very large remote one. I wrote a whole piece about that in The number that sounds right. Nothing about running the model on your own laptop repeals it. A fluent answer with no inspectable source is still just a fluent answer.
If the site doesn't publish the answer, the assistant is instructed to say so and point you at the contact page rather than filling the gap with something that sounds about right.
The website is the knowledge base
There's no separate chatbot dashboard to keep fed.
The site is Astro and the writing lives in Markdown. Each build turns the published pages into a compact search index. A new article joins the assistant's reference material because I wrote the file and published the site, not because I remembered to go and paste it somewhere else afterwards.
Autonomy includes saying no
The local model is optional because the machine belongs to you, the visitor.
If the browser can't run it, search still works. If you would rather not spend the 700 MB, search still works. If you turn it on and think better of it an hour later, there's a control that clears the cached model out of the browser.
The questions and answers exist in the current page and nowhere else. Reload and they are gone.
There is no analytics layer recording what people ask. I built this to help somebody work out whether I am any use to them, not to turn their curiosity into another thing I measure. The precise behaviour is in the privacy notice and the cookie notice.
This is definitely a worse assistant than the alternative
Which is worth saying plainly, because anyone who has used a frontier model will have thought it several paragraphs ago.
A 1B model running inside a browser tab is less capable than what you get through a hosted API. It's slower to start. It's constrained by the device, the memory and its own size. If I wired this to a commercial endpoint tomorrow it would answer better, it would answer faster and almost nobody would notice the difference in where it happened.
I still think the architecture should follow the operating boundary rather than the fashion. The knowledge base here is small, the questions are bounded and the source material is already public. In that specific case a local model gives the visitor a more private option without making the thing depend on an account, an API key or a metered service that has to keep being paid for.
Change any of those conditions and I would build it differently. Local models are not a principle. They were the right fit for this job, on this site, at this size.
And I should declare the obvious. This is a website that exists to get me hired, so a working demonstration of the thing I am claiming to be good at is not a neutral design choice. It is a considerably better argument than a page listing my skills.
The order is the part I wouldn't want to get wrong. It's also not the order that comes naturally. The usual instinct is to reach for the model first and work out the boundary afterwards, once it is built and the boundary has become a paragraph in a policy.
Consent belongs in the system, not in the write-up of it.
Autonomy is not a settings page added after the decision. It is the decision.