How it started

In 2016, a friend asked for my help with invoicing. They were invoicing manually using Excel and wanted me to improve their Excel setup. I suggested dropping Excel and built a small ERP prototype for orders and invoices. It was enough to convince them that this was the right approach.

Soon, a lot of new feature requests and ideas started to accumulate. I insisted that they start using the application in their daily work to clarify what they actually need to complete the full end-to-end process — and only after that we’ll look at other ideas.

I launched Siikli in early 2017. We quickly realized we were missing some core functionality and a lot of the nice-to-haves were eventually forgotten.

Siikli order listing view. Used to manage orders and bulk-print waybills (kuormakirjat).

Reliability learned on a Saturday morning

One Saturday morning in 2017, I woke up to multiple WhatsApp messages: Siikli was down and it was blocking their shipments since there was no way to print waybills for truck deliveries.

I quickly noticed that Tomcat had crashed. No logs, no errors, it just wasn’t running anymore. I restarted it, and they were able to continue deliveries. Clearly, I never wanted to start my Saturday like that anymore. I added a cron job that checked every five minutes that Tomcat was still running and restarted it if needed.

A few months later, I received similar messages again: This time MySQL had crashed. I added a similar script for it.

After two stressful situations, I really started to prioritize reliability. I changed daily backups to hourly backups and started copying them to two different countries. I added a notification emails if the application stopped responding.

As the system became more critical, I started to become more and more worried about security. I limited SSH access to my own IP and started hiding version numbers. I got worried about the access logs showing huge amount of crawlers looking for .env files and other vulnerabilities, so I added a firewall to only allow traffic from Finland.

These were pretty basic things, but improved the quality of my sleep.

Ownership when you’re the solo dev

I already felt that I had been taking a lot of ownership at my day work, but Siikli felt different.

When something failed, there was no one else to blame. I realized how easy it would’ve been to blame someone else for forcing me to rush to keep a deadline, which led to mistakes. No, all mistakes were mine.

What would I do differently

Siikli never became a major success. The mental overhead of maintaining a mission-critical application as a solo dev has been too taxing compared to the income I’ve made. The security, backups and reliability were all bigger headaches than I had anticipated. Even though I’ve not done any coding for Siikli in 3 years, I feel the stress is like a sub-process somewhere in my brain, taking 1-2 % of mental capacity.

After 8 years, I’m now considering either growing the app or discontinuing it. Even discontinuing an app is not easy when your users rely on it.

Why we migrated

We originally launched our servers in the US with high expectations of usage from there, but reality went the other way. We started getting users from Northern Europe, and to optimize latency and meet GDPR data residency requirements, we decided to move the whole infrastructure to EU.

Choosing the migration strategy

Most migrations require choosing between consistency and uptime. We chose to prioritize uptime.

We took a calculated risk: we would snapshot the US database while the system was running, copy the snapshot to Europe, and restore it there. This meant accepting a 60-minute window of data loss for anything written after the snapshot.

For many applications, this would be an absurd idea. For us, it made sense. Our users receive a confirmation email, and most of them won’t even notice or use their account. By migrating during the night, we estimated that the data loss would only affect 1-2 users, and they will unlikely need their accounts. Keeping the system reachable and functioning throughout the process was more important.

The actual migration

I started by duplicating all resources: two databases, two Redis instances, two ECS services — two of everything. I basically ran a warm standby in the new region.

I had never run a multi-region service before, and it was interesting to see the Terraform pieces locking into place. Terraform modules can be used multiple times with different AWS providers (from different regions). This immediately exposed a design mistake in my Terraform code: I had bundled global resources (like IAM roles) into regional modules. I had to move 37 resources to support this use case.

At 22:00, with all infrastructure and data ready, I switched CloudFront origin from the US to Europe. I had practiced this in staging, but it still felt like taking a leap of faith.

The edge case I missed

Traffic moved immediately and everything seemed fine. But a few seconds later, I saw an error in the logs: User ID does not exist.

I was too focused on complex issues so I had overlooked a simple issue: if a user created an account in the US database and we switched traffic to the 60-minute-old copy, their newly created account would vanish in the middle of the process. Fortunately, we had planned our client-side to handle unexpected errors. It kept a copy of the form data, and two minutes later I saw the user completing their process.

Takeaway

The infrastructure work was a great exercise in multi-region Terraform. We accomplished what we needed with a simple approach.

The idea

I believed that competition would motivate students. Kids who normally weren’t paying attention in class would be more eager to learn math when it was used in a competitive game. I borrowed ideas from Counter-Strike: teams, flag capturing and scoreboards. Then I used them for math learning.

The implementation

I forked BrowserQuest, an open-source game built by Mozilla. I replaced sword fights with math battles: both players saw the same question, like 10×3, and the first player to answer correctly dealt damage. The battle ended when one player ran out of health.

I divided players into red and blue teams, added a lobby, respawning, a minimap, scoreboards, and a custom map. I even built chat functionality to make it more engaging.

The tech stack

The game was built with Node.js v0.8, WebSockets and Canvas API.

The multiplayer aspect was the biggest challenge. Latency and async messaging meant that each player saw the game events differently. Since there wasn’t a centralized, authoritative server for the game logic, the server had to reconcile conflicting client messages into a shared state. You never knew which message to trust.

The classroom test

I ran the experiment at an elementary school in Turku, Finland, with 7-year old students. The students were split into two groups: a group who played the game and a control group.

Students played the game very differently than I expected: weaker students often gave up during math battles. They just did nothing and waited to lose. Because the question refreshed for both players as soon as one answered, slower students realized they had no chance and simply stopped trying.

A lot of features went unused. Minimap, chat, leveling up, bots had no use. 7-year-olds didn’t really understand flag capture and didn’t coordinate attacks in a team.

I had spent weeks, even months, building features that had no value. Something much simpler would’ve been better for learning and faster to build.

The results

The group that played the game improved their quiz result by 56%. The control group, sitting in a regular math class, improved by 87%.

Regular math class was significantly more effective than my game.

What I learned

I still think the project was cool, and there’s something respectable about committing to a crazy idea. But I should have simplified a lot.

The core idea was way too stressful. Forced competition isn’t motivation since half of the class is below average. Everyone needs to learn at their own pace.

Why it still matters

Y Combinator’s guide How to get startup ideas has a concept that fits perfectly here. It warns about “solutions in search of problems”. Taking an idea that worked, but building it for another purpose. X but for Y.

That’s exactly what I built: Counter-Strike but for math. I was so busy solving technical problems that I never stopped to ask if I was solving the real problem — learning math. Since then, I’ve learned to start from the problem.

The mistake: Trusting the Host header

We had a vulnerability in the “Forgot password” feature for admin users. When a user requested a password reset link, the application used the HTTP Host header to determine which domain (staging or production) to use in the link that was emailed to the user.

This is a classic mistake: you should never trust the Host header as it can be forged by an attacker.

By forging the header, an attacker could generate a reset link pointing to their own domain:
https://evil-domain.io/api/auth/password-reset?token=.... Since the email only showed “Reset your password” button, it wasn’t obvious that the link had been tampered with. If the admin clicked it, the secret token was sent to the attacker.

The second layer to the rescue

We had implemented a “Defense in Depth” strategy and all admin endpoints (/api/admin/*) were protected by IP whitelisting. Only traffic coming from our office VPN was allowed through.

The attacker could have successfully hijacked the token and reset the password, but they still couldn’t perform any admin actions. Their IP wasn’t on the whitelist.

During the security audit, the issue was downgraded from critical to high because of this safeguard. It was still bad, but contained.

“But IPs can be spoofed”

Technically, yes, but spoofing is practically useless in this situation:

• The attacker doesn’t know the correct IP. Guessing a specific IPv4 address is impractical, and our rate-limit makes it almost impossible.

• Spoofing breaks the response path. TCP/IP protocol works so that the server’s response goes to the spoofed IP address, not back to the attacker. So while they could send blind POST and DELETE requests without knowing if they succeeded, they wouldn’t receive any sensitive data back.

Simply: the data can’t leak. That eliminates a massive attack vector.

You need AWS Backup, not just RDS backups

Standard RDS backups are tied to the instance lifecycle. If the instance is deleted, you risk losing its automatic snapshots.

You need to use AWS Backup to decouple backups from the instance.

Compliance mode: no one can delete the backups

We’ve locked our AWS Backup Vault in compliance mode. This makes the backups immutable. They cannot be deleted: not by a tired developer, a disgruntled employee, and even an attacker with full administrator access. This is a definitive defence against ransomware.

The downside is that you can’t delete the backups even if you want to. If you accidentally generate terabytes of data, you have to pay for it until the retention period (35 days for us) expires. Even AWS support cannot bypass this. We accept that risk. I would rather risk a high bill than business continuity.

Hourly backups and RPO

We run backups every hour, the shortest interval AWS Backup supports. This gives us a Recovery Point Objective (RPO) of one hour. In a worst-case disaster, we lose 60 minutes of data.

Don’t lose a database by accident

“Deletion Protection” should be enabled on every production RDS instance. It’s a numbers game: With enough time and enough developers, the “wrong environment” mistake will eventually happen. I’ve seen it.

Disaster-proofing with cross-region replication

We replicate our backups to two regions: Stockholm (eu-north-1) and Frankfurt (eu-central-1). If an entire region goes down, our data remains safe and accessible.

For absolutely certainty, 3-2-1 strategy would be the best: 3 copies (original +2 regions), 2 different types (RDS + Backup Vault) and 1 offsite (outside of AWS). We haven’t implemented the offsite copy yet, but cross-region copy covers our current risk profile.

Cost breakdown

Let’s assume the database has 1 GB of data with moderate churn.

Restoring backups

Backups are just the beginning. Continously monitor that backup jobs are actually succeeding and test that you can restore the data.

Also, don’t forget to back up everything: secrets, configuration files, and all forms of data (like S3).