A project is never
just finished.

The idea takes shape

A home energy project, not just another dashboard.

AgilePi grew out of a simple ambition: understand what was happening with the energy in my home, keep my own records, and see whether I could make better decisions about when to use, store or import electricity.

I did not want to stop at looking at the numbers in a manufacturer's app. I wanted to collect the information myself, keep a history, combine it with Octopus Agile prices and weather forecasts, and gradually build software that could make useful decisions from all of it.

That idea became a collection of Raspberry Pi scripts, databases, services and experiments. It has taken plenty of turns since then, and not every experiment has worked first time.

The first AgilePi building blocks

Getting prices, forecasts and information onto the Pi.

The original AgilePi work brought together several small but useful jobs: collecting Octopus Agile electricity prices, collecting gas prices, creating forecasts and refreshing a compact Inky display. Each job had its own Python script, schedule and log file.

The price collector stores tariff data in an SQLite database. The forecast process developed into a wider pipeline, recording predicted half-hourly prices as well as a daily renewable-energy score. The Inky display became a way to see useful information without opening a browser.

Cron gave the little computer its routine. Price collection, gas updates, forecasting and display refreshes were scheduled independently. Systemd was used for services that needed to remain running. It was a practical, modular start: one job could be investigated without having to rebuild the whole project.

Learning to look beyond the current half-hour

Weather, renewable energy and prices become part of the picture.

The project gradually moved beyond storing today's electricity prices. A forecast pipeline was developed to look ahead at predicted half-hourly Agile prices, retain information about the source and model used, and keep forecast values separate from prices that had actually been published.

There was also a weather and renewable-energy strand. The Open-Meteo hourly forecast was used to examine wind speed and solar radiation over a seven-day horizon. The daily renewable score combined wind and solar, with wind weighted at 80% and solar at 20%. Night-time solar values were excluded from the daylight calculation so that darkness did not distort the daily solar picture.

The solar forecast was treated as an estimate of possible PV generation, not guaranteed spare energy. The controller needed to consider solar remaining today, tomorrow's forecast, household demand and battery charging losses before deciding how much grid energy might be needed.

I also investigated the Carbon Intensity API as another source of forward-looking information. That work hit response-format problems: an error response could arrive as HTML rather than JSON, and code expecting a structured object could fail when it received a string. It was a useful reminder to validate external data before letting it influence a decision.

Bringing solar and EcoFlow into the project

Four panels, a battery and a lot more data to understand.

The energy installation at the heart of the project is built around an EcoFlow Stream Ultra X battery and an EcoFlow Smart Meter. Solar comes from four panels: two JA Solar panels rated at 465 W each and two DMEGC panels rated at 450 W each.

That gives the installation a combined panel nameplate rating of 1,830 W, or 1.83 kWp. The figure is the panels' rated capacity, not a promise of what they will generate on an ordinary day. Real output depends on conditions, orientation, temperature and the limits of the system.

The challenge was to bring the individual PV inputs, battery state, battery power, grid readings and household-side measurements into one consistent picture. That meant learning how the EcoFlow API and MQTT messages represented the system, rather than simply trusting every field name at face value.

Making the system remember

Ten-second readings and the first proper history.

The EcoFlow integration developed into a set of services: an MQTT monitor to receive the data, a live JSON snapshot, a small Flask endpoint for current readings and a logger writing history into SQLite.

The intended logging interval was around ten seconds. The records included battery state of charge, battery power, solar output, grid power, individual PV inputs, remaining capacity and other telemetry. The database gave the project something the live app could not: a record of what had happened, not just what was happening now.

A captured database sample covered readings from 13 September to 17 October 2026 and had grown to roughly 238,000 rows. That was a useful milestone, but it also set up one of the next problems: a small Pi can collect data faster than it can always serve it back to a browser.

The first monitoring and remote-access headaches

A dashboard is only useful when you can reach it.

The EcoFlow monitoring pages were served locally by Flask. Cloudflare Tunnel was used to make the web interface reachable away from home through my domain. That brought its own set of practical jobs: origin-service errors, connection refusals, tunnel configuration and DNS behaviour all had to be understood and checked.

Email and domain configuration also became a small project of their own. Forwarding mail through Cloudflare and sending as my own domain worked until authentication and DMARC alignment became an issue with some recipients. The lesson was that a domain is not just a web address; mail authentication, DNS records and the return path all need to agree.

These jobs were not the main energy project, but they mattered. Remote access and reliable alerts are part of running a system that lives on a computer in the house.

A backup scare and a proper recovery check

A backup file is not necessarily a good backup.

Backup work exposed a particularly unhelpful failure mode: some SQLite backup attempts produced zero-byte files. A file with the right name is no comfort if it contains no recoverable database.

The process was reviewed so the live logger and SQLite database could be handled more carefully. A recovery copy was checked with SQLite's integrity check, which returned 'ok'. At that point the verified copy contained 221,480 readings and was uploaded to the Google Drive backup location using rclone.

There were other bumps along the way, including rclone configuration permissions and an expired token. The important change was the approach: verify the database and the remote copy before treating a backup as dependable. A scheduled task is useful, but a tested restore path is what really matters.

The optimiser gets a serious rethink

Moving away from fixed rules and towards the economics.

The controller became the most complicated part of AgilePi. The goal was not to keep the battery at an impressive percentage for its own sake. The real aim was to minimise the cost of supplying the home while using the battery sensibly.

Earlier versions had produced schedules that were too rigid, including long 0 W discharge holds overnight or into the morning. That raised an important question: a HOLD does not make energy free. When the battery is prevented from discharging, the grid has to supply the load instead. A hold only makes sense when the cost of importing now is outweighed by the value of saving battery energy for a more expensive period, with losses and any wear assumptions properly included.

The optimisation was reshaped around a rolling 72-slot horizon — 36 hours of half-hour periods — with published prices taking priority and forecasts filling gaps. The model considered solar remaining today, tomorrow's solar forecast, battery state, demand estimates and the likely economics of charging, holding or allowing normal discharge. Actual prices had to replace forecast values slot by slot as soon as the supplier published them; the controller could not assume that a whole day of prices was available just because the usual publication time had passed.

The user-defined operating limits also mattered: a 4% lower floor could be acceptable when the economics and outlook justified it, while 92% was an upper operating ceiling rather than a target. Earlier logic had used a 10% hard floor and a fixed 55% evening entry target. Those were challenged as arbitrary: the right amount of energy to carry into the evening should depend on likely demand, the remaining solar, prices and the next day's outlook.

Negative and free-price periods added another wrinkle. When the tariff is genuinely negative, it can make sense to let the grid supply the home rather than spend stored battery energy, subject to device and strategy safeguards. Conversely, charging at a full battery is not automatically a problem because the EcoFlow manages its own charge limit. The controller's job is to compare the real options, not blindly follow a clock or a target percentage.

The controller learns to say no

A missed deadline or weak data should stop a write.

Testing revealed that a clever schedule is not enough. The controller also needs fresh telemetry, adequate price coverage, a useful demand history and enough time to prepare and commit a schedule before the next half-hour boundary.

The v9.4-fixed work tightened those checks. It re-aged telemetry at the final write check, validated battery state, added minimum demand-history quality gates, checked that the writer supported explicit enable and disable flags, and accepted only the writer's documented literal Boolean True as an acknowledgement.

If production validation failed, the controller attempted to clear existing recurring charge and hold tasks rather than leave an old schedule quietly in place. That is an important safeguard, but an acknowledgement from the writer is still not a live read-back from the battery. The EcoFlow app remains the place to verify the stored device tasks.

The test suite was also strengthened. Thirteen deterministic regression tests passed in the documented v9.4-fixed test run, alongside a mocked end-to-end TEST-mode run. The model also carried explicit assumptions for charge/discharge efficiency and modest battery wear costs — including 0.35 p/kWh stored and 0.25 p/kWh delivered in the documented configuration. Those are tuning inputs, not measured battery degradation rates. The tests improved confidence in the code; they did not prove real-world savings or unattended reliability on the physical device.

The real device is a harder test than a report

Commands, acknowledgements and the difference between the two.

Production experiments brought the control path closer to the EcoFlow equipment. MQTT publish acknowledgements and reply messages were observed, but the replies were not decoded or correlated sufficiently to count as proof of the stored device state.

Some controller runs were accepted by the writer; others were refused. A run on 18 October, for example, stopped because telemetry had become stale by the final write check, even though the snapshot had been fresh when first fetched. The controller cleared the existing recurring tasks by acknowledgement, rather than proceeding with a late write.

This was frustrating, but it exposed exactly why the safety checks exist. A controller should not force a schedule through simply because it has spent several minutes calculating one. When its data or timing is no longer trustworthy, refusing to act is the better outcome.

The Raspberry Pi hits its limits

Collecting every ten seconds creates a lot of history.

The EcoFlow history database had grown to around 324 MB and approximately 325,722 rows by 27 October. The history page was becoming painfully slow on the original Pi Zero, particularly when asked to return a full day or a week of readings.

One 24-hour API request took about 49 seconds and a seven-day request took more than eight minutes in the measured test. The problem was not just the volume of data: SQLite was not consistently choosing the useful timestamp index for the query.

Adding and explicitly using the timestamp index transformed the underlying query timings. The measured indexed tests were about 0.06 seconds for one hour, 0.16 seconds for four hours and just under one second for 24 hours. That was a big improvement, although the Pi still had to serialise and send large responses and the browser still had to draw the graphs.

The work also led to a more sensible plan for retention, graph ranges and reducing the amount of data sent to the browser. The intention was to keep useful history without making the little computer struggle to display it.

Keeping the project running and recoverable

The less glamorous work: crashes, services and backups.

There were periods when the Pi crashed or rebooted and the controller stopped behaving as expected. We looked at service status, kernel messages, USB behaviour, logging and the surrounding system rather than assuming every fault came from the controller itself.

The backup arrangement was refined further. The intention was to pause only the logger when necessary for a consistent SQLite copy, take a proper online database backup, restart the logger, then archive the project and transfer it with rclone. A systemd service and timer were used for the scheduled backup rather than relying on an unverified manual job.

The project also had to be documented more carefully. Old scripts, active scripts, databases, logs and credentials are not interchangeable. The rule became simple: preserve live data, keep backups verifiable, archive uncertain old files, and never publish configuration containing secrets.

A new home for AgilePi

Moving on from the Pi Zero to a Raspberry Pi 3 B+.

The project moved onto a Raspberry Pi 3 Model B+ Rev 1.3. That gave the services and the growing collection of monitoring and control code a more capable home than the original Pi Zero.

The migrated setup retained the familiar building blocks: Python, Flask, SQLite, Mosquitto MQTT, cron and systemd. The EcoFlow live service and history service remained separate, with the live status endpoint on port 5000 and the history/API application on port 5001, both bound to the Pi's LAN address.

A practical networking quirk was worth recording: testing the Flask services through 127.0.0.1 on the Pi did not behave as expected because the applications were bound to the LAN address. Testing against the actual LAN address was the reliable check. It is a small detail, but exactly the sort of thing that can waste an evening if forgotten.

Turning the technical project into something explainable

Writing down what the system actually does.

A technical project overview was assembled to describe the hardware, data flow, software services, price and solar forecasting, controller assumptions, safety checks and outstanding work. The aim was to explain the architecture without pretending every part was already fully autonomous or proven in everyday operation.

The documentation records the four panel ratings, EcoFlow equipment, approximate ten-second data collection, 36-hour controller horizon, SQLite stores, external data sources and remote-access arrangement. It also makes an important distinction between configured assumptions and measured results: wear costs are modelling inputs, forecasts are estimates, and financial savings have not been claimed without evidence.

A separate project page was prepared for the site, with a clearer overview of the system and its technical inventory. That exercise helped identify the difference between describing a capability and claiming that it has been validated in live use.

The website starts to become a proper front end

Astro, shared navigation and a clearer project structure.

The public-facing site began taking shape as a static Astro project, developed on a Windows PC in VS Code. The aim was to give AgilePi a home that could explain the project and its hardware, software, documentation, diary and energy monitoring without exposing the private machinery behind it.

Shared header and footer components were introduced so navigation and page structure stayed consistent. The site grew to include Home, Project, Hardware, Software, Documentation, Diary, About and Energy pages. There were a few messy moments along the way — broken frontmatter imports, duplicated navigation links, a missing Energy link and an Astro configuration syntax error — but the build was repaired and the static pages generated successfully.

The Energy page then grew into a proper dashboard: live EcoFlow readings, individual PV inputs, total solar generated today, historical solar and power-flow graphs, battery state of charge and confirmed Octopus Agile prices. Household demand needed special care because the legacy `house` field was misleading; the dashboard uses the individual PV, grid and battery contribution fields for its displayed demand calculation, while the coverage of monitored circuits remains something to validate physically.

A local development proxy connected the browser to the two Flask APIs without requiring the APIs to be exposed directly to the public internet. Keeping the public static site separate from private API endpoints, logs, credentials and control functions remains an important release requirement.

The Agile graph was refined more than once. The controller plan was added using the actual per-slot decisions from the latest saved schedule, rather than guessing from prices. The visual treatment was adjusted until the charge and hold decisions could be read alongside the tariff bars without making the graph needlessly busy.

A more useful view of the controller's plan

Putting the decisions beside the prices they belong to.

The confirmed-price endpoint was refined to return one record per half-hour slot, even where the underlying database contains duplicate rows. The duplicate records were left untouched in the database; the API groups them for display. The graph can therefore show the available published prices without clutter or accidental double-counting.

A daily solar-total endpoint was also added. It integrates the recorded PV1–PV4 power readings since midnight using the trapezoidal method and skips gaps longer than 60 seconds, rather than pretending that missing intervals contain valid measurements. The dashboard can show the result in kWh and Wh alongside the instantaneous PV output.

A separate schedule endpoint exposes the controller's latest saved per-slot actions. The website matches those decisions to the exact half-hour price slots. GRID_CHARGE and HOLD_0W are displayed only when the controller slot is marked confirmed and published; normal operation is left unmarked.

The plan panel also reports the controller mode and its last writer acknowledgement. The wording deliberately explains that an acknowledgement is not a live read-back from the EcoFlow device. This is a small but important example of the project's approach: make the data useful, but do not overstate what it proves.