Typolino – Upload Lesson

Finally I find some time to write about the upload utility. Maybe we can add an online editor later, but to get started quickly I decided to use a pure node application running on my local device using the Firebase Admin SDK.

Now we are already connected to our DB and storage and can just upload and manipulate the data as we need. Each lesson will be store in a folder containing the actual lesson structure and all the media files.

And this is an example of a single lesson:

The lesson.json file has the same content as we will upload to the DB.

{
  "name": "Blumen",
  "description": "Lerne die Blumen kennen",

  "difficulty": "EASY",
  "words": [
    {
      "word": "Rose",
      "sentence": "",
      "imageId": "rose.jpg",
      "audioId": "rose.mp4"
    },
    {
      "word": "Tulpe",
      "sentence": ".",
      "imageId": "tulpe.jpg",
      "audioId": "tulpe.mp4"
    },
    {
      "word": "Lilie",
      "sentence": "",
      "imageId": "lilie.jpg",
      "audioId": "lilie.mp4"
    },
    {
      "word": "Seerose",
      "sentence": "",
      "imageId": "seerose.jpg",
      "audioId": "seerose.mp4"
    },
    {
      "word": "Sonnenblume",
      "sentence": "",
      "imageId": "sonnenblume.jpg",
      "audioId": "sonnenblume.mp4"
    }
  ]
}

Note I just configured the audioId, but we are not using it yet. So.. you can just ignore it for now.

Upload Script

The upload script is really simple and can be used like:

node upload.js <folder>

It will look for the lesson.json file, create a lesson document in DB and upload all non JSON files to the storage. It expects you have no typos in the configuration – but it’s not that hard to fix it and just upload again.

We will discuss this topic later and (hopefully) even improve more. As with many cloud services you have a pay-as-you-go model (OpEx) it is important to minimize these costs as much as possible. So having to download multi MB images and then just render them in a small 640px * 480px resolution doesn’t make much sense. I love to tune things as much as possible, I feel being back at the times where resources where sparse and finding creative ways to save some money is really cool. 🙂 One aspect where we definitley need to look into later on is caching.

Long story short: we transform every image before uploading to fit our target resolution. For this I found a cool library called Jimp.

(async() => { 
    const admin = require("firebase-admin");
    const fs = require("fs");
    const Jimp = require("jimp");

    const arguments = process.argv;
    const lessonFolder = arguments[2]; // find a better way e.g. oclif?

    console.log(`You would like to uplaod folder ${lessonFolder}`);

    const serviceAccount = require("C:/Dev/typolino-firebase-adminsdk-4dg10-b2047407d9.json");
    const app = admin.initializeApp({
    credential: admin.credential.cert(serviceAccount),
    databaseURL: "https://typolino.firebaseio.com",
    });

    // upload the data
    const lesson = require(`./${lessonFolder}/lesson.json`);
    const firestore = app.firestore();
    const storage = app.storage();
    const bucket = storage.bucket("gs://typolino.appspot.com/");

    // create basic structure
    firestore.collection("lessons").doc(lessonFolder).set(lesson);

    // upload all files
    fs.readdirSync(lessonFolder)
    .filter((file) => !file.endsWith(".json"))
    .forEach(async (file) => {
        const imagePath = `${lessonFolder}/${file}`;
        const image = await Jimp.read(imagePath);
        await image.resize(640, 480);
        await image.quality(80);        
        bucket.file(imagePath).createWriteStream().end(await image.getBufferAsync("image/jpeg"));                
    });
})();

Not much tuning yet, but at least we don’t upload super big files. That’s all! For now it is good enough and does the job.