Mafalda SFU

Mafalda SFU receives “Best Scalable Real-Time Media Platform 2026” at the Spanish Business Awards

2026-02-05T00:00:00+00:00

We are happy to share that Mafalda SFU has been recognised with the award “Best Scalable Real-Time Media Platform 2026” at the Spanish Business Awards organised by EU Business News.

The official listing can be found here: https://www.eubusinessnews.com/winners/mafalda-sfu/

While Mafalda SFU started as an experimental project exploring scalable real-time media architectures, it has gradually evolved into a platform that demonstrates how modern infrastructure can support high-performance, distributed, real-time communications.

What Mafalda SFU is about

Mafalda SFU is a Selective Forwarding Unit (SFU) designed to support scalable real-time media applications built on technologies such as WebRTC.

Instead of mixing or transcoding streams centrally, an SFU forwards media packets between participants, allowing applications to scale efficiently while keeping latency extremely low.

The project focuses on several key ideas:

Horizontal scalability for large multi-participant sessions
Low-latency media routing for interactive experiences
Modular architecture that allows integration with different signaling systems
Efficient bandwidth usage through selective forwarding

These characteristics make SFU architectures particularly suitable for:

video conferencing platforms
real-time collaboration tools
live interactive streaming
multiplayer experiences and virtual events

Why scalability matters in real-time media

Real-time media systems are fundamentally different from traditional web applications.

They must deal with constraints such as:

strict latency requirements
packet loss and network variability
adaptive bitrate and congestion control
large numbers of concurrent participants

Traditional MCU-based architectures (multipoint control units) often struggle to scale efficiently because they require expensive media mixing and transcoding.

SFU architectures instead focus on intelligent packet routing, enabling platforms to handle many participants with significantly lower compute requirements.

Mafalda SFU explores how this model can be implemented in a modern, distributed infrastructure while keeping the system lightweight and flexible.

An experimental platform exploring real-time media architectures

From the beginning, Mafalda SFU was conceived less as a commercial product and more as a technical exploration of scalable media systems.

The project investigates questions such as:

how to design efficient SFU routing layers
how signaling systems interact with media routing
how distributed deployments can support large sessions
how WebRTC infrastructures can be simplified while remaining scalable

These explorations are increasingly relevant as real-time media becomes a core component of modern platforms, from communication tools to collaborative and immersive applications.

Recognition from the Spanish Business Awards

Receiving the Best Scalable Real-Time Media Platform 2026 award highlights the growing importance of scalable communication infrastructure in today’s digital ecosystem.

As remote collaboration, live media and real-time interaction continue to expand, architectures like SFUs play a crucial role in enabling new kinds of online experiences.

This recognition reinforces the relevance of continued experimentation and research in this area.

Looking forward

Real-time media infrastructure is still evolving rapidly.

Projects like Mafalda SFU contribute to exploring new architectural approaches that can help shape the next generation of communication platforms.

While the project remains experimental, the ideas behind it of scalability, efficiency and modularity in real-time media systems will continue to influence how modern communication platforms are designed.

Learn more

More information about the award can be found here: https://www.eubusinessnews.com/winners/mafalda-sfu/

If you are interested in real-time media architectures, SFUs, or scalable WebRTC infrastructure, stay tuned for future updates and technical articles.

How to scale EduMeet (or any Mediasoup application) with Mafalda SFU

2025-03-31T00:00:00+00:00

A frequently asked question we get is how to scale an already existing Mediasoup-based application with Mafalda SFU. From an application or user perspective, the scalability is transparent, because Mafalda SFU handle all the complexity of scaling the application; and from the developer perspective, the API is the same as with a single Mediasoup instance, so there’s no need to learn anything new. The only difference is that the application has to use any of the Mafalda SFU packages instead of directly using the Mediasoup package. This is a pretty simple change, and it can be done for any Mediasoup-based project in a few lines of code, as can be seen in the example of our Remote Mediasoup client mock repository. But to make things easier, we’ll have a few examples in this post about porting EduMeet to Mafalda SFU about how to use and combine the different Mafalda SFU packages to scale your application. You can also check the code changes explained in this post already applied to EduMeet for each different use case in our EduMeet fork, with some additional fixes and updates.

Prepare your application

Although Mafalda SFU is designed to be 100% API compatible with Mediasoup, they have a different architecture: meanwhile Mediasoup directly export its functions at module level, being effectively like a singleton object, Mafalda SFU modules export a manager class, that is responsible of implementing the actual functionality of each Mafalda SFU module, and later they provide a Mediasoup compatible object instance, that’s the one that actually replaces the Mediasoup module.

This means, the first thing you need to do in your project code is to replace the Mediasoup module import, and require it as a function argument, allowing to do dependency injection, so later we can pass the Mediasoup compatible object instance instead. This is a pretty simple change, and it can be done in a few lines of code, being the only difficulty in case the Mediasoup module is being imported from multiple files. In that case, you need to change them to be imported from a single file, and pass the Mediasoup object instance to the different files, like in this example:

import mediasoup from 'mediasoup';

- async function run()
+ async function run(mediasoup)
{
  const worker = await mediasoup.createWorker()

  // ...
}

- run()
+ run(mediasoup)

You can see a real world example at our EduMeet fork.

Once you have your application ready to use dependency injection, next step is to replace the Mediasoup object instance with the Mediasoup-compatible one from the Mafalda SFU package you are using.

The basics: Remote Mediasoup client

The most simple way to scale a Mediasoup based application using Mafalda SFU is to use the Remote Mediasoup client package. This package allow to connect to a Remote Mediasoup server instance, that can be running on a different (and potentially more performant) server. This is the recommended way to use Mafalda SFU, because it’s the most flexible alternative allowing to fully abstract your application from the underlying Mediasoup implementation, and depending on the application needs, you can increase the media servers capacity independently of the application servers, specially when connecting to Mediasoup cluster or Mediasoup vertical cluster servers. This way, you can have a single application server that can make use and handle multiple Mediasoup servers at the same time and scale them independently.

To replace Mediasoup with the Remote Mediasoup client, you only need to install the Remote Mediasoup client package, and then replace the Mediasoup object instance with the Remote Mediasoup client and its Mediasoup-compatible object instance. If you have the Remote Mediasoup client package tarfile at the root of your project, you can replace the Mediasoup package with it with the following commands:

npm uninstall mediasoup

npm install ./mafalda-sfu-remote-mediasoup-client-*.tgz

After that, you can replace the Mediasoup object instance with the Remote Mediasoup client and the Mediasoup compatible object instance:

+ import { once } from 'node:events';

- import mediasoup from 'mediasoup';
+ import RemoteMediasoupClient from '@mafalda-sfu/remote-mediasoup-client';
+
+ const client = new RemoteMediasoupClient('ws://example.com');

+ // Destroy the client when the process is terminated, to do a clean exit
+ const onProcessSignal = client.destroy.bind(client);
+
+ process.once('SIGINT', onProcessSignal);
+ process.once('SIGTERM', onProcessSignal);

+ function onConnectionFailure(error)
+ {
+   console.error('Error connecting to Remote Mediasoup server:', error);
+
+   process.exit(1);
+ }
+
+ // Wait for the connection to be established, or notify connection error
+ await once(client, 'mediasoup').catch(onConnectionFailure);

+ // The connection is established, we can use the Mediasoup compatible object
+ // instance, and listen for errors
+ const { mediasoup } = client.on('error', console.error);

(Sorry if you find the diff hard to read, GitHub don’t support code syntax highlighting for diffs. I’ve split each part with white lines to make it easier to spot and follow them.)

You can see a real world example at our EduMeet fork.

Remote Mediasoup client mock

If you just want to to check how much difficult is to port an existing Mediasoup based application to Mafalda SFU, you can use the Remote Mediasoup client mock package. This package allows to use the same API as the Remote Mediasoup client, but instead it’s implemented as a thin wrapper around the Mediasoup package, so you can use it to validate that Mafalda SFU is a good fit for you, or test your application in local during development without having to install and configure a separate server.

To use it, it’s pretty similar to the Remote Mediasoup client, you just need to replace the Mediasoup package with the Remote Mediasoup client mock one, and having the same API on purpose, the code changes would be the same:

npm uninstall mediasoup

npm install @mafalda-sfu/remote-mediasoup-client-mock

+import { once } from 'node:events';

- import mediasoup from 'mediasoup';
+ import RemoteMediasoupClientMock from '@mafalda-sfu/remote-mediasoup-client-mock';

+ function onConnectionFailure(error)
+ {
+   console.error('Error connecting to Remote Mediasoup server:', error);
+
+   process.exit(1);
+ }

+ // URL is not really used, but it's required for compatibility
+ const client = new RemoteMediasoupClientMock('ws://example.com');
+
+ const onProcessSignal = client.destroy.bind(client);
+
+ process.once('SIGINT', onProcessSignal);
+ process.once('SIGTERM', onProcessSignal);
+
+ await once(client, 'mediasoup').catch(onConnectionFailure);;
+
+ const { mediasoup } = client.on('error', console.error);

You can see a real world example at our EduMeet fork.

Scale up: Mediasoup Vertical

If your scalability needs are not that high, and you don’t need to scale your application over several servers, you can scale it to use all the CPUs cores at once, using the Mediasoup Vertical package. This package aggregates the performance of all the CPUs cores of a server as if they were a single one, allowing to have sessions beyond the participants capacity of each CPU core.

Using it is similar to the Remote Mediasoup client, just only you need to provide a Mediasoup Manager object as argument. In its simplest form, it’s just an object with a mediasoup property, that is the Mediasoup object instance, and a getStats method, that returns the global and system stats where the Mediasoup instance is running. In case you want to run Mediasoup Vertical in local on top of an instance of the original Mediasoup package, you can use the function provided by the Mediasoup getStats factory package:

# We don't need to uninstall the Mediasoup package, since we'll use it

npm install @mafalda-sfu/mediasoup-getstats-factory
npm install ./mafalda-sfu-mediasoup-vertical-*.tgz

Then, you can use it like this:

- import mediasoup from 'mediasoup';
+ import getStatsFactory from '@mafalda-sfu/mediasoup-getstats-factory';
+ import MediasoupVertical from '@mafalda-sfu/mediasoup-vertical';
+ import mediasoupOrig from 'mediasoup';

+ const { close, getStats } = await getStatsFactory(mediasoupOrig);
+
+ const mediasoupVertical = new MediasoupVertical(
+   { getStats, mediasoup: mediasoupOrig }
+ )
+   .on('error', console.error);
+
+ function onProcessSignal()
+ {
+   mediasoupVertical.destroy();
+   close();
+ }
+
+ process.once('SIGINT', onProcessSignal);
+ process.once('SIGTERM', onProcessSignal);
+
+ // No need to wait for the connection to be established, original Mediasoup
+ // package object is always available
+
+ const { mediasoup } = mediasoupVertical;

The same as before, you can see a full real world example at our EduMeet fork.

Scale out: Mediasoup Horizontal

The same way that Mediasoup Vertical works as a wrapper around a Mediasoup Manager to aggregate its Workers performance, the Mediasoup Horizontal package can be seen as a wrapper around (multiple) Remote Mediasoup client instances, where it manages their connection status and the media routing between their Worker instances, as if all of them were in a single Mediasoup instance.

In this case, we create multiple Remote Mediasoup client instances, each one to connect to a different Remote Mediasoup server, and later we provide them to Mediasoup Horizontal to do its management:

npm uninstall mediasoup

npm install ./mafalda-sfu-mediasoup-horizontal-*.tgz
npm install ./mafalda-sfu-remote-mediasoup-client-*.tgz

+ import { once } from 'node:events';

- import mediasoup from 'mediasoup';
+ import MediasoupHorizontal from '@mafalda-sfu/mediasoup-horizontal';
+ import RemoteMediasoupClient from '@mafalda-sfu/remote-mediasoup-client';

+ function newClient(url)
+ {
+    // We disable `listenWorkerDied` to prevent duplicated notifications
+   return new RemoteMediasoupClient(url, { listenWorkerDied: false });
+ }
+
+ function onConnectionFailure(error)
+ {
+   console.error('Error connecting to Remote Mediasoup servers:', error);
+
+   process.exit(1);
+ }
+
+ function onProcessSignal()
+ {
+   // We first remove (and destroy) the clients, to do proper cleanup tasks
+   for (const client of clients) client.destroy();
+
+   mediasoupHorizontal.destroy();
+ }

+ const urls = [
+   'ws://example1.com',
+   'ws://example2.com',
+   'ws://example3.com'
+ ];

+ const clients = urls.map(newClient);
+
+ const mediasoupHorizontal = new MediasoupHorizontal(clients);
+
+ process.once('SIGINT', onProcessSignal);
+ process.once('SIGTERM', onProcessSignal);
+
+ await once(mediasoupHorizontal, 'mediasoup').catch(onConnectionFailure);
+
+ const { mediasoup } = mediasoupHorizontal.on('error', console.error);

You can find the full example at our EduMeet fork.

Full scale: combined scalability

Finally, the most versatil (not counting on using Mediasoup vertical cluster package) and complex use case is to combine and use both Mediasoup Vertical and Mediasoup Horizontal packages at the same time. This allows you to scale your streaming sessions over multiple CPU cores running on several servers, having your capacity only limited by how many servers you have available. But at the same time, it’s easy to implement it thanks to the Mafalda SFU packages, because they are designed to be compatible and combinable between them.

Similar to the Mediasoup Horizontal use case, you need to create multiple Remote Mediasoup client instances and pass them to the Mediasoup Horizontal instance, but in this case, you also need to create a Mediasoup Vertical that will be initialized with the Mediasoup Horizontal instance, stacking the Mediasoup Vertical instance on top of the Mediasoup Horizontal one. This way, the Mediasoup Horizontal instance will be responsible of collecting the Workers from the multiple Remote Mediasoup servers, and the Mediasoup Vertical one will aggregate them to make them work as a single one:

npm uninstall mediasoup

npm install ./mafalda-sfu-mediasoup-horizontal-*.tgz
npm install ./mafalda-sfu-mediasoup-vertical-*.tgz
npm install ./mafalda-sfu-remote-mediasoup-client-*.tgz

+ import { once } from 'node:events';

- import mediasoup from 'mediasoup';
+ import MediasoupHorizontal from '@mafalda-sfu/mediasoup-horizontal';
+ import MediasoupVertical from '@mafalda-sfu/mediasoup-vertical';
+ import RemoteMediasoupClient from '@mafalda-sfu/remote-mediasoup-client';

+ // We disable `listenWorkerDied` to prevent duplicated notifications from both
+ // Mediasoup Horizontal and the Remote Mediasoup clients
+ const options = { listenWorkerDied: false };
+
+ function newClient(url)
+ {
+   return new RemoteMediasoupClient(url, options);
+ }
+
+ function onConnectionFailure(error)
+ {
+   console.error('Error connecting to Remote Mediasoup servers:', error);
+
+   process.exit(1);
+ }
+
+ function onProcessSignal()
+ {
+   for (const client of clients) client.destroy();
+
+   // No need to destroy the Mediasoup Vertical instance, it will be destroyed
+   // automatically when the Mediasoup Horizontal instance is destroyed
+   mediasoupHorizontal.destroy();
+ }

+ const urls = [
+   'ws://example1.com',
+   'ws://example2.com',
+   'ws://example3.com'
+ ];

+ const clients = urls.map(newClient);
+
+ const mediasoupHorizontal = new MediasoupHorizontal(clients, options);
+ const mediasoupVertical = new MediasoupVertical(mediasoupHorizontal);
+
+ process.once('SIGINT', onProcessSignal);
+ process.once('SIGTERM', onProcessSignal);
+
+ await once(mediasoupVertical, 'mediasoup').catch(onConnectionFailure);
+
+ const { mediasoup } = mediasoupVertical.on('error', console.error);

The final full example can be found at our EduMeet fork.

Mafalda SFU perpetual beta (AKA 1.0)

2024-04-01T00:00:00+00:00

After exactly three years of development, lots of awake nights (and sunshines), and uncountable hours of debugging and testing, we are proud to announce today, April 1st 2024, that we have completed Mediasoup aggregated scalability and stability tests, and we are ready to publicly release the first feature complete version of Mafalda SFU. In other words, at last Mafalda SFU has got “1.0 version” status 🥳

This has been a huge milestone, so between highs and lows during these last three years (including some hiatus periods due to professional or personal reasons), we have finally reached the point where we can say that Mafalda SFU is ready for any kind of production use case, being that by having lots of small sized rooms each one under a single CPU capacity limits where horizontal scalability is the best fit, one or several medium sized rooms making use of several CPUs from a single server (vertical scalability), up to having one or several big sized rooms extending over multiple servers where aggregated scalability shines. Also, the high modularity of Mafalda SFU (at this moment we have 11 main projects, and 73 repositories on GitHub between public and privates ones) allows us to adapt to any kind of use case and architecture, and to evolve over time with your future requirements.

Now you would be asking, “why are you calling it ‘perpetual beta’ if you are releasing a 1.0 version?” Well, there are three reasons for that (three is a magic number 🎶):

First of all, as part of our commitment to be 100% API compatible with Mediasoup, we automatically check three times a day for new Mediasoup releases and pass all our project tests against them (and also, we pass all Mafalda SFU projects against Mediasoup tests themselves) for each new version, so instead of following semantic versioning, we make and publish a new release of all Mafalda SFU projects from their git main branches every time a new Mediasoup version is released, with all both Mediasoup and Mafalda SFU features and bug fixes up to that date. This way, the version number warrants the compatibility of each Mafalda SFU package with the exact same Mediasoup version.
We are not going to stop developing Mafalda SFU: we have a lot of ideas not only about how to improve Mafalda SFU reliability and performance (and they are already pretty high!), but also about new features and projects in mind that we want to implement, like monitoring tools, offering Mafalda/Mediasoup as a Service (MaaS) in addition to our current on-premises approach, or even decentralized P2P streaming on top of Mediasoup over multiple regions in the world.
And finally, we are going to keep the “perpetual beta” status as a reminder that we are always looking for feedback, and we are always open to new ideas and improvements, so if you have any suggestion, idea, or you want to collaborate with us, please don’t hesitate to contact us.

And if you are questioning yourself: no, this is not an April Fools’ joke, we are really releasing Mafalda SFU 1.0 today, we just wanted to make you think it 😉 So, if you want to know more about Mafalda SFU, please visit our website, or if you want to contact us, please write us to info@mafalda.io.

How to migrate from Jest to node:test

2023-04-24T00:00:00+00:00

Jest is one of the most populars testing frameworks for Javascript and Node.js. Originally developed by Facebook, it’s a one-stop-shop with testing, assertions, code coverage… but this implies some critics, like requiring more than 50mb of dependencies. Also, somewhat recently was shown to be maintained mostly by a single person, being that the reason why updates and maintenance was so much slow, so they decided to transfer it to OpenJS foundation. Also there has been several long standing critics about not providing a pure environment, or the fact that Jest parses the code, leading to some complexities when needing to configure transpilation. That has lead to several people looking for alternatives, and having now a built-in test runner in Node.js, I decided to see myself how to migrate to it.

First of all, it’s to identify the replacements for the different Jest features. As already shown, node:test module can the used as the test runner and for mocking, meanwhile assert module can mostly be used for assertions. A tough topic are inline snapshots, since it’s a feature I use a lot. Luckily Jest snapshot engine is provided as an independent module, but it don’t have documentation at all and it’s very tightly coupled with Jest itself. There are some modules that wrap it and add support for chai like chai-jest-snapshot and mocha-chai-jest-snapshot, so it can works as a solution, but not only that would add an extra assertions library, but also there’s no support for inline snapshots, just only regular ones. I tried to create my own wrapper without making use of any extra dependency and adding the support for inline snapshots, but lack of documentation makes it difficult. Maybe I’ll try again in the future (anybody interested on sponsors me? :-) ).

Having now the elements, doing the migration is surprisingly pretty straighforward:

Add chai and chai-jest-snapshot as dev dependencies, and remove any Jest related dependency and configuration.
Rename __tests__ folder(s) to test.
Replace jest command with node --test.

Add the node:test functions that previously were set as globals by Jest:

import {
  afterEach, before, beforeEach, describe, mock, test, throws
} from 'node:test'

Enable the snapshot support, and the replacement of the expect() function:

import chai, {expect} from 'chai'
import chaiJestSnapshot from 'chai-jest-snapshot'


chai.use(chaiJestSnapshot)

before(function()
{
  chaiJestSnapshot.resetSnapshotRegistry()
})

Since node:test run each test in a different process, we need to add the snapshot support in each test file.

Comment or remove calls to expect.assertions(), since there’s no equivalent in node:test. This was possible on Jest since assertions were integrated in the tests runner.
Await the Promises instead of expect() on them (expect(await promise) instead of await expect(promise).resolves).
Replace Jest idioms:
- expect(func).toThrowErrorMatchingInlineSnapshot(error) with throws(func, error)
- expect(value).toBe(expected) with deepStrictEqual(value, expected)
- expect(value).toBeInstanceOf(class) with ok(value instanceof class)
- expect(value).toBeUndefined() with ifError(value)
- expect(value).toMatchSnapshot() with expect(value).to.matchSnapshot()

Missing points as already commented, would be the support for inline snapshots, and maybe create a codemods to automate the process :-)

Previously published at https://piranna.github.io/2023/04/24/How-to-migrate-from-Jest-to-node-test/

Profiling `npm install` times

2023-02-05T00:00:00+00:00

When installing Mafalda SFU packets, a problem I’ve suffered several times are install times, specially since I’m using git dependencies. I tried to reduce times by publishing some of the most common packages to npm, so removing need to install and compile development dependencies like Typescript, but still install times were huge for no reason, so I needed some way to measure the install time of each one of the dependencies. This lead out options like UNIX time command or tools like slow-deps, so just by change, I found on StackOverflow a reference to gnomon.

gnomon just only prepend timestamp information to the standard output of another command, so it’s a bit crude for my use case, but with some tweaking can provide the needed info. In this case, by running npm install in verbose mode, we can get detailed info of each one of the steps it follows, so we can measure its time delay. Just only, npm install provides that info in stderr (non sense, what is using stdout for?), so we need to redirect it first so gnomon can get it on its stdin:

npm install --verbose 2>&1 | gnomon

This will show the output in two columns, being the timestamps in the left one. On average, it looks like 90% takes less than 1 second and 99% less than 10 seconds… but there’s a 1% of outliers that get up to 1 minute or more. Focusing on that ones, we find two of them (output is truncated for brevity):

...
  12.8811s   npm http fetch GET 200 https://registry.npmjs.org/@jscpd%2fhtml-reporter 2117ms (cache revalidated)
   0.2557s   npm http fetch GET 200 https://registry.npmjs.org/ws 258ms (cache revalidated)
...
  59.4045s   npm http fetch GET 200 https://codeload.github.com/Mafalda-SFU/on-change/tar.gz/718eda3ad6c777f7f8df08908603aab8e9f1082e 1325ms (cache revalidated)
   1.1857s   npm info run bcrypt@5.0.1 install node_modules/bcrypt node-pre-gyp install --fallback-to-build
...

gnomon show the time each line has been the last one shown in the terminal, so that time correspond to the generation of the next line. This is particularly useful when showing “starting…” lines, to know in what task this time has been employed so much, but when showing only the conclusions (like it’s the case), we need to focus on the next line instead.

Total time, 206.7054s, whom 12 are spent on downloading ws package, and a full minute compiling bcrypt instead of using a prebuild image. It’s a bit too much, but since the total time is just a bit over 3 minutes, and the problem is probably due to use Node.js v19.4.0 instead of a LTS version, I would not worry too much.

Another project I have been having more important problems with install time has been the Remote Mediasoup integration tests, where install of all Mafalda SFU subprojects is involved, and in fact Github Actions cancel the job step after about 10-12 minutes (does it has a timeout, or it thinks I’m mining cryptocurrencies?). So checking with gnomon, the timings percentages seems similar, just only we found the outliers got rampage:

...
  34.4881s   npm verb logfile /home/piranna/.npm/_logs/2023-02-05T11_55_57_787Z-debug-0.log
   0.2060s   npm http fetch GET 200 https://registry.npmjs.org/@mafalda%2feslint-config 2345ms (cache revalidated)
...
  13.4045s   npm http fetch GET 200 https://registry.npmjs.org/@piranna%2frpc 1680ms (cache revalidated)
   0.1576s   npm http fetch GET 200 https://registry.npmjs.org/@xmldom%2fxmldom 227ms (cache revalidated)
...
 743.5718s   npm http fetch GET 200 https://codeload.github.com/Mafalda-SFU/multi-map/tar.gz/08f018c637bd1261558fecaf3998e099298dca14 1340ms (cache revalidated)
   0.0409s   npm info run @mafalda/mediasoup@3.11.8 install node_modules/@mafalda/mediasoup node npm-scripts.js install
...
 188.7822s   npm info run @mafalda/rope-client@0.0.0 postinstall { code: 0, signal: null }
   0.1286s   npm info run mediasoup@3.10.5 postinstall { code: 0, signal: null }

Total time, 1059.6964s (18 minutes), whom of them 34 seconds are spent on downloading Mafalda SFU eslint-config package (hum? Why so much?), and 15 minutes compiling both Mafalda SFU custom Mediasoup prebuilds and the regular one (there are some incompatibilities starting on version 3.10.6 that prevent using the latest one with Mafalda SFU, working on fixing them at this moment).

Mediasoup, what else?

Mediasoup by design is compiled on destination platform at install time, so it can take a lot of time to install with all CPU cores at 100% and the machine totally unresponsive (and I have a laptop with 20 CPU cores…). That’s why I’ve tried in the past and now again to generate and publish automated prebuilds of Mediasoup with a nightly Github Action. I hope to make them work properly with newer versions of the prebuild images.

In any case, my internet connection is not very fast or stable, so it’s not surprising that it takes so long to download all the packages or there are so much differences between them, and results are not conclusive but provides a good guidance. Probably with a better internet connection, it could show the bottleneck is somewhere else, probably at the compilation of packages. In addition to that, a good improvement for gnomon or a similar tool would be to add statistics about CPU load, to see what are the installed packages that put more pressure on the install process.

Previously published at https://piranna.github.io/2023/02/05/Profiling-npm-install-times/

How to use private repositories as npm git dependencies on Github Actions

2022-12-25T00:00:00+00:00

I’m advocate of automatization, and that includes not only CI/CD pipelines, but also I wanted to do it for documentation publishing. Mafalda SFU is split in a lot of packages (currently more than 30!), so I wanted to have a single place where to publish the documentation of all of them. Github Pages allows to host a website for your organization or username by free (this blog and personal site already makes use of it), and it can also host automatically a website for each repository as sub-paths of your username/organization main website. Problem is, that it only works for open source repositories or for paid plans, and most of the Mafalda SFU repositories are private ones. So since the Mafalda SFU project website is already hosted on Github Pages as a public repository, I decided to store and serve from it all the other repositories documentation as well… doing it in an automated way :-)

The task can be splitted in two halves: generate the documentation, and publish it.

TL;DR

This is the Github Actions workflow I’m using to tests the different Mafalda SFU sub-projects, generate their documentation, and publish it to the Mafalda SFU project website:

name: test, build documentation and publish

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
      - uses: webfactory/ssh-agent@v0.7.0
        with:
          ssh-private-key: $

      - run: npm ci
      - run: npm test
      - run: npm run docs

      - uses: cpina/github-action-push-to-another-repository@v1.5.1
        env:
          SSH_DEPLOY_KEY: $
        with:
          destination-github-username: Mafalda-SFU
          destination-repository-name: Mafalda-SFU.github.io
          source-directory: docs
          target-directory: docs/$
          user-email: bot@mafalda.io
          user-name: Mafalda bot

Generate the documentation

To automatically generate the documentation, we have two aproaches: generate it as a git hook, or do it on the CI servers. I use to prefer the first option, specially for complex projects where people of different levels are working on the code (like Juniors & Seniors, or new employees), or for the projects where the documentation is going to be directly published on Github Pages (like open source projects) so the documentation is close to the code as reference, and also it’s possible to see how it has evolved during the time. But since it’s not possible to publish it as I’ve already said it before (if not, I would not be writing this post :-) ), it has had more weight my policy of not having generated code or artifacts stored as part of the source code of the git repository itself, so I decided to generate the documentation on the CI servers.

To generate the documentation, I’m using the jsdoc-to-markdown package. This imply to need to install the project dependencies on the CI servers, and this is where the problem arises: since I have not published any of the Mafalda SFU dependencies as a npm package, I need to use them as git dependencies, and being private ones, the Github Actions CI servers needs permissions to access to them. In normal conditions where only it’s accessed the repo itself, or when using private packages from the Github Packages Registry, it’s enough with the GITHUB_TOKEN secret or with a Personal Access Token, but since we’re using git dependencies, it’s not possible to use them to authenticate the git requests, only possible is with git credentials and/or an SSH key pair.

The first thing I’ve done is to create a new machine user account to authenticate and operate from Github Actions. This is for security, I could have used my own account, but then all operations would have been done in my name. This way, it’s possible to have a separate user with the lowest needed permissions (in this case just only read access to the private repositories, that’s the one set when adding an user as one of the organization members by default), and also it’s possible to revoke the access to the repositories if needed. After that, I’ve created a new SSH key pair, and added the public key to the Github machine user account I’ve just created before, so it can operate on the repositories

After that, what we need to do is to add the private key to the Github Actions, so it can be available when trying to install the git dependencies. This can be done with the ssh-agent action, that registers globally the provided private key. To do so, we need to set the private key as one of the repository secrets (it’s not possible to add it as an organization level secret on free plans), and then use it in the config of the ssh-agent action in the ssh-private-key parameter. Just with that, we can be able to install the npm git dependencies from Github Actions CI servers, and generate our documentation.

Publish the documentation

Once that we have the documentation generated, it’s time to copy it to the main repository. This is done with the github-action-push-to-another-repository action, that can push the contents of a directory to another repository different of the one where the action is running on. The config of the action is pretty straighforward, just setting the source directory from where to copy the files, the destination repository name, the Github username/organization that owns the destination repository, and the target directory where to copy them (this is important, by default it fully wipes the target repository). Just by config that parameters it would work, but commits would be mostly annonimous, so it’s better to also set the user name and user email that will be used to commit the changes.

The only tricky part of configuring github-action-push-to-another-repository is the access to the repository itself, since we need to provide another credentials with write access. The most safe way to do it is to create a new SSH deploy key that will be used to provide write access only on the destination repository, in this case the main documentation one. This is done by creating a new SSH key pair, and adding the public key to the destination repository as a deploy key with write access. Then, we need to add the private key as one of the repository secrets, and use it in the config of the action in the SSH_DEPLOY_KEY. After that, each time that the action is executed, it will push the generated documentation to the main repository, creating a new commit with the user email and name that we provided, and a commit message pointing to the original commit in the source repository where we generated the documentation from, having this way a two-ways cross-reference.

Previously published at https://piranna.github.io/2022/12/25/How-to-use-private-repositories-as-npm-git-dependencies-on-Github-Actions/

Presenting Mediasoup Horizontal

2022-01-02T00:00:00+00:00

Although Mafalda SFU is mainly focused on vertical scaling of Mediasoup and the WebRTC stack, the main problem I’ve found companies are facing is about how to easily implement Medisoup horizontal scaling. I’ve been working on a solution for this problem for a while on, and since Mafalda SFU is build on top of Mediasoup, it’s also needed to help it to provide transparent vertical and horizontal scaling, so let’s see how it works.

Mediasoup-Horizontal is a manager that allows to remotely control multiple instances of Mediasoup, providing a simple and easy to use API based on the one from the Javascript Set object. It allows to add and remove Remote Mediasoup Client objects and manage them, although is not directly tied to them so you can provide your own objects following that same API, for example to also control a local Mediasoup instance in addition to the remote ones.

When adding a Remote Mediasoup Client object to the Mediasoup-Horizontal instance, it checks that all the Remote Mediasoup instances are compatible between them, so you don’t have to worry about in what instance your Mediasoup objects are being created. After that, it monitors the creation of new objects, and also does it with the Client current objects in case they are later destroyed.

Mediasoup-Horizontal also provides a Mediasoup compatible API, so by using it you can auto-magically enable your application to scale horizontally without needing to change your current code. The “magic” happens by using objects that provides the same API of Mediasoup Worker and Router classes, but internally proxying the method calls to their internal Connection objects, shared by all the Client instances currently connected to the same server. This is done this way to be transparent to the actual objects references being done at Mediasoup-Horizontal level.

After that, implementation of pipeToRouter() method is fairly trivial. Code is an optimized version of original Mediasoup one, just only with fine-grain errors management and with some performance optimizations to reduce delay of events propagation, so general behaviour is the same, although there’s space for improvements. In fact, I proposed to move it out officially to a separate library so it can be reused by other projects like this one, but the proposal was rejected.

And that’s it, that’s how Mediasoup-Horizontal works. Next steps I’m planning about are to improve performance and resilience, allowing it to better recover from network re-connections, and also works towards implementing a tool for monitoring in real time the status of all the Remote Mediasoup instances.

Previously published at https://piranna.github.io/2022/01/02/Presenting-Mediasoup-Horizontal/

WebRTC horizontal scaling

2021-09-26T00:00:00+00:00

When approaching the horizontal scaling of WebRTC servers, we have two main approaches: decentralized P2P, and using a central server. Each one has its own drawbacks and advantages, and I had difficulties to identify what approach was the best, since I usually have a personal preference for pure P2P architectures, but they are not the most simple nor always the more efficient ones. So when deciding how to approach Mafalda horizontal scaling, I needed to consider the pros and cons of each use case I would need, and here we have my conclusions.

P2P

The first one needs to maintain its own list of servers where to ask for extra resources. This list can be provided when starting the server, but if the extra servers can be added and removed dynamically, this list will need to be updated later, so we need a way to update the list. In addition to that, when querying for another server with enought free resources to connect to, since we don’t have a central place to store the info, we need to ask to all the servers to get their updated state so we can apply the heuristics to decide which one to use, or the servers would need to send their state info to the other ones at intervals. In both cases, if the number of servers is large, the bandwidth needed to send the state info to all the servers will be high. One solution to this would be to use a DHT or a database, but in both cases we would need anyway to download the info to do the heuristics, and the second one leads us to a centralized solution.

Central server

So instead of having a central database where to store the state of the servers, and need to query it to get the state of the servers, we can have instead central server to do the queries for us. This central server would became a single point of failure in case it gets down, so we can have mutiple instances and store the state info of the servers in a database. Seems a bit like walking in circles, but this one can be close to that query servers, so there would not be so much bandwidth issues if they are located in the same place.

This central server (or servers) would just receive a request from the WebRTC servers asking to connect to another empty enought server to increase their own resources, and do all the heuristics to decide which one to use on its own, or also decide to automatically spin-up a new server if there’s no available ones and it’s configured to do it, so all the logic would be centralized in a single place and being transparent to the other ones. It would do the request for the state of the WebRTC servers and cache it, or receive the updates from them at intervals, probably using an unreliable WebSocket connection or any other similar one since the info is ephimeral and there’s no need to keep it in sync all the time.

In conclussion, between these two approaches, it seems that the central server is the one that best fit this particular use case, since it’s the one that best balance CPU and bandwidth usage, and also allow to easily change the heuristics without needing to spin-down the WebRTC servers.

Update: Mafalda SFU aproach

Based on these two traditional network architectures, I decided to apply my own solutions for each one of the use cases, the centralized and decentralized one.

Mafalda-horizontal

Mafalda-horizontal package is build on top of Remote Mafalda, and the same than this one or the original Mafalda one, it follows the same Mafalda API, so it’s intercambiable with them and allow easily to upgrade from the vertical scalability aproach to the horizontal one without modifying the code. It works as a client to multiple Remote Mafalda instances, managing the balancing of the resources, and also the routing between the servers. Also, since it follows the Mafalda API, it can be used as a Remote Mafalda instance, so it’s possible to implement an hierarchical architecture with multiple layers of servers, all of them accesible from a single endpoint, while at the same time not wasting resources to propagate the clients media between the servers since these ones gets connected directly to the instances of Mediasoup.

Mafalda-swarm

On the other hand, Mafalda-swarm package implements a decentralized federated P2P aproach, and in fact this is the one I originally envisioned to implement, being Mafalda-horizontal just an intermediate step to help me to develop the remote versions of Mediasoup and Mafalda modules… and have quickly an horizontal scalling solution that can be of commercial interest while I develop Mafalda-swarm, that’s the really interesting and funny one to implement :-P

In Mediasoup, the most important info to consume a stream is its Producer ID. These ones are unique UUIDs, and they can be explicitly set by hand. In fact, they are the only ones Mediasoup objects whom constructor allow it, and the Mediasoup method Router.pipeToRouter() sets the Producer in the destination Router with the ID in the original one, so it can be considered a good practice. Based on that, Mafalda-swarm makes use of a DHT to map the Producer IDs to the servers where’s located the original one, so it’s possible to directly ask to it and decrease lags and delays. The original producer server can be busy, so it will delegate the request to another server from the ones that are already consuming that stream, so this way they can balance themselves without external administration, just adding new servers to the swarm.

Previously published at https://piranna.github.io/2021/09/26/WebRTC-horizontal-scaling/

Presenting Mafalda SFU

2021-07-22T00:00:00+00:00

Mafalda SFU is a massively vertical and horizontal scalable SFU built on top of Mediasoup. This allow to have (theorically) unlimited sized WebRTC-based video room calls.

Mediasoup is one of the most important SFUs: open source, performant, and easy to use, but it’s too much low level and RTP streams oriented, so it’s main issue is its lack of off-the-shelf scalability. Due to that, Mediasoup Router instances are limited to a single CPU core and a reduced number of participants.

During to the Covid pandemic, I’ve been working extensively in WebRTC projects in the last year and a half, and one of the topics that most times surfaced was how to use Mediasoup with big video rooms in both an easy to use and performant way. So, after finding some issues in the way other projects were aproaching this problem, I started to think about my own solution.

Mafalda aproach is simple: just create a MafaldaRouter instance, and it will scale itself to use all the server CPU cores and create the connections to other servers. From both developers and users PoV, it’s like using a single Mediasoup Router instance, but running in a server with an humongous big CPU. Not only that, but since Mafalda schedule algorythm try to minimize the number of used connections and CPUs, you can run multiple MafaldaRouter instances in the same server, and Mafalda will balance the resources between them. And when they are not needed anymore, connections get closed and the CPU cores are freed, so you can stop the servers to reduce costs, all this without needing to administer them and almost without configuration.

The main features of Mafalda are:

Fully automated: you only need to create a MafaldaRouter instance, and it will take care of everything else
Optimiced resources consumption on half-used machines compared to libraries using other architectures, allowing to host multiple rooms on the same machine without needing to pre-assign resources, that could lead to a suboptimal usage
Designed to support adding a new server to the cluster, without needing to restart the whole cluster. It also can make use automatically of new added CPU cores in a server (if it has support) without needing to restart the process
API heavily influenced by Mediasoup, so it’s easy to understand, use and migrate from your existing code
100% acceptance tests code coverage for both lines, branches statements and functions. This has shown to be very useful to find bugs until the very last moment

I’ve been working on Mafalda since March 2021, and so far I’ve fully implemented vertical scaling, and got to a point where both API and code are stable and ready to start testing in real environments.

At the same time, I’m currently working on horizontal scaling too. So far I’ve implemented a client library with the same API of local Mafalda instances, and designed an events-based RPC protocol that replicate the state of the remote Mediasoup objects and the Mafalda instances, allowing to work with them as if they were local ones. That would help not only to easily migrate deployments from local to cluster environments, but also to simplify automated management of horizontal scaling and inter-server connections. More on that in a future article ;-)

Previously published at https://piranna.github.io/2021/07/22/Presenting-Mafalda-SFU/