I Was Spending 5--10 Minutes Entering Every Event Flyer. So I Built an Agent.

One of the things I've learned running Events in Vacaville is that a lot of local event information doesn't live in a database, calendar, or API.

It lives on flyers.

A restaurant posts a trivia night to Instagram.

An organizer posts a concert flyer on Facebook.

A local business announces a workshop with an image containing everything you need to know.

As a person, that's great. I can look at the flyer and immediately understand what's happening.

As the person trying to maintain a structured local events website, it's a little more complicated.

For a while, my workflow looked like this:

Find flyer → take screenshot → read flyer → manually create structured JSON → review → ingest

It took me about 5--10 minutes per event.

That's not terrible for one flyer.

But 20 flyers?

That's 100--200 minutes.

Thirty?

Potentially five hours of copying information out of images and turning it into structured data.

That wasn't how I wanted to spend my time.

So I started wondering if I could get a machine to do the boring part.

I wasn't trying to remove myself from the process

This distinction is important.

I wasn't trying to build an autonomous system that would crawl social media, decide what belongs on Events in Vacaville, extract everything and publish it without me.

I still wanted to make the editorial decision.

When I'm using the EIV social accounts and see an event that fits what we share, I take a screenshot and email it.

That's my trigger.

The system can handle what happens next.

Today, an agent polls the mailbox, looks for a keyword in the subject line, extracts the attached flyer and processes it.

When it's finished, I get a Slack message.

I review the result.

The goal wasn't:

How do I remove the human?

It was:

How do I make the human part take five minutes instead of several hours?

That turned out to be an interesting AI engineering problem.

I already had the hardware

I'd previously given a presentation to my team about running LLMs locally, and I had a Mac mini that I was already using for local AI experimentation.

This seemed like a good opportunity to put that knowledge to work on an actual problem.

The first version of the agent used:

Flue Framework + Ollama + qwen3-vl:30b

Everything ran locally on the Mac mini.

Qwen's vision capabilities meant I could give it an event flyer and ask it to transform what it saw into the schema Events in Vacaville expected.

And it worked.

Sort of.

The output often looked convincing.

But I quickly discovered that "this looks pretty good" isn't a particularly useful metric.

A model can read the flyer and still get the event wrong

Some of Qwen's mistakes were straightforward extraction problems.

It would confuse the venue.

It would occasionally miss a date that was clearly visible on the flyer.

But one of the more interesting failures was the event description.

I'd give it a flyer expecting something like:

Join a local winery for an evening of live music, food and wine...

Instead, the model would sometimes give me something conceptually closer to:

The flyer has a purple background with white lettering and an image of a wine glass...

Technically, the model had understood the image.

It just hadn't understood my job for it.

I didn't want a description of the flyer.

I wanted it to derive what the event was from the flyer.

That distinction eventually became important.

"Accuracy" was too vague to be useful

I wanted to improve the agent, but before I could do that I needed a better answer to a basic question:

What does "good" mean?

So I built an evaluation set.

I selected 24 real event flyers.

They weren't all from Vacaville intentionally. I didn't want to optimize the system around the design patterns of a handful of local organizers. I wanted it to become better at understanding event flyers generally.

For every flyer, I manually created a golden JSON representing what I considered the correct structured output.

Then I could run the agent against all 24 flyers and compare its output with mine.

But I didn't just score the entire JSON as right or wrong.

I scored the individual fields.

Things like:

• event name
• organizer
• venue/location
• start date
• start time
• end time
• price
• whether the event represented a series
• description

That gave me something much more useful than a single accuracy number.

If dates were performing well but venues weren't, I knew what to work on next.

Not every mistake costs the same

I also didn't weight every field equally.

A date has a defined answer.

So does a start time.

A venue.

An organizer.

A price.

Those fields matter a lot because getting them wrong can mean publishing incorrect event information.

Descriptions are different.

There's legitimate room for creative interpretation there, so I weighted description quality lower.

As long as the description actually described the event rather than the image, and it saved me editing time, it was doing something useful.

That scoring system represented something important about the project:

I wasn't measuring the model against some abstract definition of AI quality. I was measuring it against the work I needed it to do.

And that led to another rule.

Null was not a failure

Not every flyer contains every piece of information in my event schema.

Maybe there's no end time.

Maybe the flyer doesn't list a price.

That's okay.

So I deliberately did not penalize the model for returning null when information wasn't available.

I'd much rather get:

"end_time": null

than:

"end_time": "9:00 PM"

when the flyer never said the event ended at 9 PM.

The first creates a little bit of work later. My ingestion/review workflow can flag the missing information and I can fix it.

The second creates something much more dangerous:

bad data that looks complete.

That meant invented values deserved special attention in the eval.

A null is cheap.

A hallucination isn't.

Then I started using autoresearch

Around this time I'd watched a talk about Andrej Karpathy's autoresearch work.

The basic idea immediately made me think of the flyer agent.

I now had exactly what I needed for systematic experimentation:

real inputs + known-good outputs + a scoring system.

Instead of changing a prompt, trying three flyers and deciding whether the result "felt better," I could test changes against the same 24 real examples.

The first run evaluated the complete JSON and established the baseline accuracy of every field.

Then I defined the priorities in a rule.md.

The fields with the greatest impact on the EIV workflow received the greatest weight.

From there, I stopped trying to improve everything at once.

We attacked one field at a time.

If startDate needed improvement, I'd set up an autoresearch run with roughly four turns focused on startDate.

Run the experiment.

Score all 24 examples.

Keep an improvement.

Try again.

Then move to the next high-value field.

Instead of:

Make my agent better.

the problem became:

Improve this measurable behavior without breaking the things that already work.

That worked.

Until it didn't.

At some point, stop blaming the prompt

After about two days of experimentation with Qwen, a pattern started emerging.

An autoresearch run might find an improvement on turn two.

Great.

Turn three would make another adjustment and the score would get worse.

Revert.

Turn four would try another approach.

Worse again.

We started seeing variations of that pattern across many of the fields.

Sometimes an instruction looked perfectly reasonable, but the model would simply ignore it.

We had reached about 79% weighted accuracy, and additional prompt changes weren't reliably moving us forward.

At that point I had to consider something I hadn't really wanted to change.

Maybe the prompt wasn't the bottleneck anymore.

Maybe the model was.

I didn't really want to change models

Changing a prompt is relatively cheap.

Changing the model underneath the entire experiment is a much bigger variable.

I wanted to be reasonably confident we'd exhausted the current approach before doing that.

But after watching the same plateau appear across multiple fields, I decided it was worth testing another model.

I had been reading good things about Gemma's vision capabilities and decided to experiment with:

gemma-4-31b-it-mlx-4bit

Initially, I wanted to keep the rest of my stack as similar as possible.

That didn't work out.

My Ollama integration wasn't giving me the image-payload workflow I needed for this model.

So now I had another problem to solve.

The requirements pushed me toward oMLX

I started looking at alternatives that made sense for Apple Silicon.

oMLX caught my attention for a few practical reasons.

It supported the image use case I needed.

It allowed me to pass configuration parameters over the HTTP request---things like temperature and, eventually, thinking configuration.

And its documented first-token performance looked attractive compared with the runtime I was currently using.

So the next configuration became:

Mac mini + oMLX + gemma-4-31b-it-mlx-4bit

This wasn't a clean model-only A/B test.

I'd changed both the model and inference runtime, so I can't honestly say that Gemma alone was responsible for whatever happened next.

What I could do was run the new configuration through exactly the same evaluation process.

So that's what I did.

79% became 92%

With Gemma running through oMLX and thinking disabled, I repeated the autoresearch process.

Same flyers.

Same golden JSON.

Same scoring philosophy.

Same emphasis on the fields that actually mattered to the workflow.

The result:

92% weighted accuracy.

The structured fields were now performing extremely well.

Dates.

Times.

Venue.

Organizer.

Price.

Most of the things where there was a defined correct answer were essentially where I wanted them.

I could have stopped there.

But there was still one field bothering me.

The description.

What if I let the model think?

The descriptions were better, but I wondered whether enabling thinking would improve the model's ability to synthesize the information on the flyer into something that sounded like an event description rather than an extraction.

There was an obvious risk.

Thinking might produce better descriptions.

It might also make the model more creative in places where I didn't want creativity.

Would it start inventing prices?

Adding times?

Inferring details that weren't actually on the flyer?

There was another tradeoff too: inference would take longer.

Claude warned me about that while I was working through the experiment.

But I realized I didn't particularly care.

The agent runs asynchronously.

I'm not sitting in front of my computer waiting for each flyer to finish.

It can take its time.

I wasn't optimizing for how fast the computer finished. I was optimizing for how little time I had to spend fixing its work.

So I turned thinking on.

The slower configuration won

The difference was significant.

With thinking disabled:

92% accuracy
~39 seconds median per flyer

With thinking enabled:

98% accuracy
~156 seconds median per flyer

The computer became roughly four times slower.

And I preferred it.

Because machine latency and human latency are not the same thing.

If this were an interactive chatbot where I was staring at a spinner for two and a half minutes, 156 seconds would be terrible.

That's not my workflow.

The agent is running in the background.

It can spend those extra minutes thinking while I'm doing something else.

When the batch is finished, Slack tells me my attention is needed.

The extra inference time costs me essentially nothing.

Cleaning up bad output does.

So the "slower" configuration was actually the faster solution to the problem I cared about.

The metric that mattered most wasn't 98%

Going from 79% to 98% was satisfying.

Watching invented values drop was even more important.

But neither number was really why I built the system.

Remember the original problem.

One flyer took me approximately 5--10 minutes to manually turn into clean structured event data.

So:

20 flyers = ~100--200 minutes

30 flyers = ~150--300 minutes

Now I can send 20 or 30 flyers into the workflow and spend roughly five minutes of my own time reviewing the results.

That's the metric I care about.

Not tokens per second.

Not benchmark rankings.

Not whether the agent can operate completely autonomously.

Human time.

The workflow today

The final system is surprisingly mundane from my perspective.

While I'm using the Events in Vacaville social accounts, I might see an event posted by a local organizer.

If it's relevant to EIV:

I take a screenshot.

I email it.

And I'm done for the moment.

The agent watches the inbox for the appropriate subject keyword, extracts the image, processes the flyer locally on the Mac mini and creates the structured event data.

When it's ready:

Slack tells me.

I review the result.

Any missing fields can be caught by the ingestion workflow.

I make whatever corrections are necessary.

Then the event can continue through the normal EIV publishing process.

The machine handles the transcription and preparation.

I remain responsible for judgment.

What I learned

The most useful part of this project wasn't discovering that one local model scored better than another.

Models will change.

Runtimes will change.

Six months from now I may be using something completely different.

The more durable lessons were about how I approached the problem.

Define success around the workflow

"Accurate" wasn't specific enough.

I needed to define which fields mattered, how much they mattered, and what kinds of mistakes actually created work for me.

Measure before optimizing

The 24 golden examples turned prompt experimentation into something measurable.

Without them, I would have been judging changes mostly by feel.

Break vague problems into smaller measurable ones

"Improve flyer extraction" is difficult to act on.

"Improve startDate accuracy across these 24 examples" is something you can experiment against.

A missing answer can be better than a confident wrong answer

For this workflow:

null > invented data

Completeness isn't always quality.

Know when you've hit the limit of the layer you're optimizing

Eventually, more prompt changes stopped helping Qwen consistently.

Instead of endlessly rewriting instructions, I changed the underlying system.

Optimize the resource that's actually scarce

My Mac mini has time.

I don't.

I'll happily let a model spend another two minutes processing something if doing so means I spend less time correcting it later.

Automation doesn't have to mean autonomy

The agent doesn't decide what belongs on Events in Vacaville.

It doesn't publish events by itself.

It removes the repetitive work surrounding the decisions I still want to make.

And that's probably the lesson from this project that matters most to me.

I didn't build the Flyer Intake agent because I wanted an AI agent.

I built it because I wanted five hours of my weekend back.