Firestore: How to migrate data between accounts/projects

I recently had to migrate data from Cloud Firestore between Firebase accounts and projects. I really enjoyed the experience and decided to share with you how to do this process that involves dump and restore the data.

It is worth mentioning that this content will also be very useful in managing backups. Since Firestore does not provide an automated backup on the Console. But calm this can be done via Google Cloud Function, however this is a subject for another post.

Just one more detail. To do these operations, the project must have a defined billing account linked to the projects. Follow the below link to see how to activate.

Cloud Billing documentation | Google Cloud

So let’s start our journey by knowing the tools we are going to use.

Step 1: Getting to know the tools

To proceed with this operation we will basically use two command line tools from Google Cloud, which are:

• gcloud — The gcloud command line interface is the main CLI tool for creating and managing Google Cloud resources. You can use this tool to perform many common platform tasks from the command line or in scripts and other automations.
• gsutil — gsutil is a Python application that allows you to access Cloud Storage from the command line. You can use gsutil to perform a wide range of bucket and object management tasks.

Step 2: The projects

The projects used in this example will be present in the same account, but the process is similar for different accounts. The only difference is that the account will have to be exchanged between the export and the import of the data. This detail for when there are two different accounts is explained at the end of this post.

The projects used in the example are: firestore-course-462bc and happy-todolist. Project IDs will be important in defining BUCKET. BUCKET ?? Calm down, we’ll see that later.

The data from the source database is shown below:

Image 1 — Origin database

Image 2 — Target database

Step 3: Exporting the data with gcloud

With gcloud already installed on the machine, you will have to authenticate to take advantage of the possibilities that the CLI gives. To do this, run the command:

gcloud auth login

After that you need to define the project you will be working on, which is the source project, where the data is, and you can do this with the command:

gcloud config set project PROJECT_ID

Where PROJECT_ID is the Google Cloud project id, which is the same as Firebase. In our case we will place our original project which in our case is gcloud config set project firestore-course-462bc

With the project defined, we can now export the data. The command to be executed is as follows:

gcloud firestore export gs://[BUCKET]

In general, a BUCKET is space in the Storage that has a series of objects and in it we can define rules for access, sharing, etc.

The Firebase project has a Bucket defined as follows: PROJECT_ID.appspot.com. In our case: firestore-course-462bc.appspot.com.

Running the above command the export would already work. However it would be loose in the bucket as shown in the image below:

Image 3 — Exported data on Storage

The ideal would be to organize it in a folder that we can call exported. Our BUCKET will be as follows gs://firestore-course-462bc/exported. Finally, our command to export will be the one shown below:

gcloud firestore export gs://firestore-course-462bc.appspot.com/exported

Let's see our Storage again:

Image 4 — Exported data in a folder

NOTE: This command exports the entire database. If you wanted a specific collection add the collection-ids option (Ex: — collection-ids = users, flowers).

To view the bucket access the Google Cloud Console, select the project, in the menu access the Storage and enter the project bucket.

Downloading data from Bucket to the machine

It is at this point that we will use the gsutil tool. With it we will download the data exported from Firestore. We need to do this to send to the target project’s Storage.

With the command gsutil cp we can copy the objects from the bucket to the machine. Just pass the address of the bucket and the folder where the data will be saved, as shown in the command below.

gsutil cp -r gs://firestore-course-462bc.appspot.com/exported ./

The -r flag indicates a recursive copy used to retrieve files from folders and subfolders, that is, everything.

After executing the command, a folder with the name exported will be created at the destination (in our case the location of the terminal ./).

We have the dump on our machine, now we are going to import it into the target project.

Step 4: Importing data with gcloud

This is the time when, if the target project is in another account, you will need to authenticate to the target account with the command gcloud auth login. In the destination account, the process will be the same as follows.

Using gsutilwe will send the data downloaded in the previous step to the target project. In our case the project is happy-todolistand the command will be like the one shown below:

gsutil cp -r ./exported gs://happy-todolist.appspot.com/exported

The command in essence is the same, only the paths that are reversed. First the local path and then the bucket path.

After executing the command, a folder named exported will be created in the destination bucket.

This name is important because in the export it is defined in the dump files. Something like exported.overralexportdata.

Now let’s go back to using gcloud. This time we are going to import into the target project and for that we need to have the project defined.

Then with the command gcloud config set project happy-todolist we will define the current project.

After that just do the import with the command below.

gcloud firestore import gs://happy-todolist.appspot.com/exported

Remembering that the same goes for exports. If you want to import a specific collection, use the--collection-idsoption.

Our target database looks like this:

Image 5 — Target database with imported data

Note that the collection that already existed in the database is maintained.

Summary

That’s it folks. With these concepts in mind you will be able to dump and restore the Cloud Firestore database. This will give you the ability to work with data backups and migrations when needed.

Please clap 👏👏 this post if it was useful to you. Thanks for reading.

References

• My YouTube channel — https://youtube.com/edigleyssonsilva
• My Github — https://github.com/geeksilva97
• Firebase Docs — https://firebase.google.com/docs/firestore/manage-data/export-import
• Schedule exports do periodic backups — https://firebase.google.com/docs/firestore/solutions/schedule-export
• The icon in the thumb was created by https://www.flaticon.com/authors/ultimatearm