How to stream and store files on KaiOS
Background
Although KaiOS is based on Firefox, unlike modern smartphones or desktops, KaiOS phones are quite resource constrained. All KaiOS-powered devices have either 256MB or 512MB RAM for the entire operating system. That memory gets allocated for core system services like telephony, audio, and graphics. Only a fraction of that memory is available to your app.
As a developer, that means reducing your app’s memory footprint to avoid the dreaded Out of Memory Error. One area where memory allocation is most direct and measureable is downloads, namely downloads done using XMLHttpRequest (XHR) and fetch.
During early testing of PodLP, users would report the app crashed when downloading popular podcasts like The Joe Rogan Experience (JRE) prior to it becoming a Spotify Exclusive. The JRE is very long-format, typically 2-4 hours. This results in MP3 files that are hundreds of megabytes. Even if the KaiOS device has an external SD Card with sufficient storage, PodLP would crash because it attempted to download the entire file as a Blob.
If you search for tutorials on downloads using JavaScript, you ultimately find code snippets that looks like this:
1const req = new XMLHttpRequest();
2req.open("GET", "/audio.mp3", true);
3req.responseType = "blob";
4
5req.onload = (event) => {
6 const blob = req.response;
7 // ...
8};
9
10req.send();
However, this example will crash on KaiOS if “audio.mp3” is sufficiently large. A precise cutoff is difficult to provide, but generally that’s roughly 20-40MB depending on the device.
Downloads, the Right Way
Fortunately, KaiOS provides several ways to download large remote resources: chunking and streaming. On KaiOS 2.5 and older versions of Firefox, there are two special XHR response types: moz-chunked-arraybuffer and moz-chunked-text. Instead of providing the entire response object during the load event, these response types will return “chunks” of varying sizes during the progress event.
1const req = new XMLHttpRequest();
2req.open('GET', "/audio.mp3", true);
3req.responseType = 'moz-chunked-arraybuffer';
4
5let bytesReceived = 0;
6let total = 0;
7let percent = 0;
8
9req.onprogress = (event) => {
10 const chunk = event.target.response;
11
12 // Calculate percent progress from byte lengths
13 total = event.total;
14 bytesReceived += chunk.byteLength;
15 percent = bytesReceived / total;
16
17 // ...
18};
Using this example, it’s now possible to download large audio or video files in “chunks,” without running out of memory. This can be combined with the systemXHR permission to download large files without running into Same Origins either!
While moz-chunked-arraybuffer was removed on KaiOS 3.0, similar and more standard approach is available
using readable streams and fetch.
1fetch("/audio.mp3")
2 .then((response) => {
3 const reader = response.body.getReader();
4 const total = Number.parseInt(response.headers.get('Content-Length'), 10);
5
6 let bytesReceived = 0;
7 let progress = 0;
8
9 return reader.read()
10 .then(({ done, value }) => {
11 bytesReceived += value.length;
12 progress = bytesReceived / total;
13
14 // ...
15 })
16 })
17 .catch((err) => /* TODO */);
This example accomplishes the same outcome as moz-chunked-arraybuffer but uses fetch readable streams to process large assets in smaller pieces of varying sizes.
Note: For both XHR and fetch it’s possible that the Content-Length header isn’t present. In this case, the total size may not be known in advance.
Storing Chunks
Although streaming and chunking solve one problem (downloading large files using limited memory), they create another: storing and accessing those chunks. Developers can still use localStorage and indexedDB, but neither supports an “append” operation. This means files would need to be stored and accessed in chunks, which can work for timeseries data like audio or video, but is certainly difficult to work with!
Caution: Certain non-standard interfaces like
IDBMutableFile do not actually work on KaiOS. Accessing these resources throws a FileHandleInactiveError because the necessary internals were never implemented.
The easiest way to store files in chunks on KaiOS is using the
DeviceStorage API. DeviceStorage allows apps to request access to store “music,” “pictures,” “videos,” or generic data on “sdcard” via the navigator.getDeviceStorage() function.
Note: Media stored using named storage will show up in default system apps. i.e. “pictures” are available in Gallery, “music” is available in Music. KaiOS does not support
.nomedia files.
Permission must first be requested in manifest.webapp (or manifest.webmanifest) for the navigator.getDeviceStorage() functions to be present. Each permission has a required "access" property that can be one of "readonly" for read-only access or "readwrite" for read & write access.
1{
2 "name": "My App",
3 "description": "Awesome App",
4 "permissions": {
5 "device-storage:videos": {
6 "access": "readonly"
7 },
8 "device-storage:pictures": {
9 "access": "readwrite"
10 }
11 }
12}
For both hosted web and privileged apps, the first time you call navigator.getDeviceStorage() it will display a permission request dialog. It is important to provide context to the user beforehand, because if permission is denied the user has to go to Settings > Security > App Permissions to change it.
Once the user has granted permission, you can now add, modify, and retrieve files stored locally. This works will with chunked downloads. For instance, here’s an example on KaiOS 2.5 for saving each chunk locally.
1const req = new XMLHttpRequest();
2req.open('GET', "/audio.mp3", true);
3req.responseType = 'moz-chunked-arraybuffer';
4
5let bytesReceived = 0;
6let music = navigator.getDeviceStorage("music");
7
8req.onprogress = (event) => {
9 const blob = new Blob([ event.target.response ], { type: 'audio/mp3' });
10
11 if (bytesReceived === 0) {
12 music.addNamed(blob, "audio.mp3");
13 } else {
14 music.appendNamed(blob, "audio.mp3");
15 }
16
17 bytesReceived += chunk.byteLength;
18};
Caution: This example is an oversimplification. Since progress events are asynchronous, a more robust solution would be to queue writes to DeviceStorage to avoid writing chunks out of order.
The best part is that retrieval is very simple! The get function returns a File for a given name. This can then be passed to URL.createObjectURL to create an object URL to use as the source for <img>, <audio>, and <video> elements.
1let music = navigator.getDeviceStorage("music");
2let request = music.get("audio.mp3");
3
4request.onsuccess = () => {
5 let file = request.result;
6 let objectURL = URL.createObjectURL(file);
7
8 audioEl.src = objectURL;
9};
Conclusion
Although KaiOS is built on Firefox OS, it runs on resource constrained hardware. Developers need to pay attention to how much memory their apps use to avoid freezing and crashing. If you find these nuances challenging and need support developing performant and reliable experiences on KaiOS, contact the author from the About page.