Writing my solution for offline storage part 1— You shouldn’t do it as I did
Most of my journey as a software developer was writing JavaScript code and I sense the NIH (Not Invented Here) syndrome is far away stronger in the JS community. Every day a new library comes out to solve any problem. Some are useful like https://www.npmjs.com/package/is-even.
So yeah, maybe as you that are reading this article, I tried to write my JQuery and my CSS framework (and failed miserably) but in fact, I never thought I would create my kinda Storage Engine. Spoiler: I didn’t – at least not as I should have.
This post starts a series of essays on my team’s journey implementing offline first in an Ionic app.
Cover image from: https://www.blog.bixlabs.com/post/offline-first-mobile-apps
Offline what?
I like the definition in the Android docs.
An offline-first app is an app that is able to perform all, or a critical subset of its core functionality without access to the internet. That is, it can perform some or all of its business logic offline.
I was working on a small startup. We had an amazing application that helps students out in their daily studies. The app creates a good distribution based on knowledge level, which exam the student is being prepared for, and so on.
We needed to have a way to make it possible for the users to study in the offline mode to avoid distractions from Tittok or something like that. We had other business reasons for that. Indeed, the users were asking for that and we do what the user wants.
Where we were
As mentioned our mobile application was a hybrid one written in Ionic. The backend was written in NodeJS, basically writing serverless functions with AWS Lambdas. For the database we used MongoDB.
Between our app and the AWS per si, there was a BaaS (Backend as a Service). That provided a combination between Parse Platform and AWS. It brings some facilities like authentication, file storage, kind of ORM, and so on. It seems to be amazing but there’s no free lunch, it brings some problems, like being less flexible.
In summary, that’s how things were architected.
Our app was very simple in terms of architecture even though we had a base code with a lot of technical debits.
In terms of features we had the following:
- cycle — the users could create their study cycles;
- study registration — our users could register their studies and have statistics about them;
- timer for studying — the user could start a timer for a specific subject and could set an alarm to be alerted when the time is up.
We had other features but these are crucial for this article’s purposes.
Start thinking about offline
Then we started to think about our offline implementation. That looked good and simple at first. We had a TypeScript code that is just a JavaScript superset and JavaScript is good with JSON. And that’s what a MongoDB document is right? It’s a JSON.
It doesn’t seem to be a bad assumption at first but as we dig deeper into our decisions we’ll see that it can lead to many problems (as it did).
The majority of our entities were supposed to be offline first. The most important of them is a collection we can call statistics. We had other collections but that was far away crucial. Because that collection was supposed to contain information on how much time was spent on a specific subject. Also which kind of study was made, whether or not it involved a test and how many questions we answered correctly in such a test, and so on.
As a young professional started thinking that it would be good for our users and not that hard for implementing.
Have you ever conjectured about an offline-first implementation? Well, I can recall my thoughts on that. I was like: “Cool, it’s just a matter of saving data in the local device, and then, when they are connected, just send it to the server”.
My plan was:
- the code only reads from the local storage (let’s do some JSON.stringify in the objects);
- when data is fetched, update the local storage;
- all the writes are made in local storage first;
- send to the server when having a connection.
You know, looking back it still seems to be a good plan, because of some of these ideas I got from how Firebase does for Cloud Firestore. But that’s the problem it’s far away from being simple.
The main questions on that: Synchronization and Local Storage efficiency. We’ll be talking about these challenges in future posts but let me summarize them.
Local Storage efficiency
After handling the synchronization, that happened. We failed in measuring the efficiency and we choose bad data structures for that. When we are working on a mobile device it can be hard to predict how different devices will behave due to differences in Operating Systems and device architectures.
The problems we faced:
- bad performance in searching for specific data
- bad performance in searching a range of data
- aggregation calculations
- saving data in the local storage
Synchronization
Our first enemy was the stronger one. We would need to handle synchronization, we would need to answer that questions:
- when the data is sent to the server?
- how do we know what data we should keep? the current or the incoming one? (conflicts)
- how to handle deletions?
Summary
We were a team of a SINGLE developer (we were in the process of contracting one more), we didn’t have good technical leadership, and we were in a hurry (looks like that’s the startup way). So we made a lot of bad decisions, and I classify the bigger one as not experimenting as much as we should have.
There are many different ways of solving the same problem. We should’ve dug deeper into the approaches and compared the results.
I’m not supposed to blame my stakeholders. You’ll see, in future posts, that some mistakes are exclusively technical. This series of articles is supposed to be a journal of bad decisions and I hope it’s useful for someone who is facing similar issues.
That’s all. See ya!
