Skip to content

WebRTC Insertable Streams After Decoder: Extract Timestamp

Oliver Hargreaves Jan 4, 2024 2:21:45 PM

In this blog, we will show you another way you can modify incoming remote video streams. I will not be mutating the video frames in this example; instead, I will be logging the frame timestamps from the incoming video streams. You can replace the logging logic with your frame mutation logic.

To begin, let’s consider what we will be doing. Since we have custom logic we want to apply to each downstream connection, we want to create a function that takes a remote connection and applies our logic to it.

let openDownstream = async (remoteConnectionInfo, layoutManager, channel) => {

The first step is to establish our remote video object.

// create a remote media object to represent the remote feed

var remoteMedia = new fm.liveswitch.RemoteMedia();

Following that, we need to define a stream object that will represent the decoded remote feed.

Once we have these two objects, we can define our connection object to represent the downstream connection.

// create a video stream object from the remote media object

let videoStream = new fm.liveswitch.VideoStream(remoteMedia);
// establish the downstream connection and pass in your object to ensure they get connected to the remote feed
let connection = channel.createSfuDownstreamConnection(

Next, we need to add some handlers to make sure we address the different negative states that our connection could enter into.

// add a handler to address changes to the downstream connection

connection.addOnStateChange(function (connection) {
  // is the connection is closing or is failing to be established, remove the remote video feed from the layout and cleanup the objects
  if ( connection.getState() == fm.liveswitch.ConnectionState.Closing || connection.getState() == fm.liveswitch.ConnectionState.Failing ) {
    // if the connection has failed try to reopen it
  } else if (connection.getState() == fm.liveswitch.ConnectionState.Failed) {
    openDownstream(remoteConnectionInfo, layoutManager);

With all of our connection logic in place, we can now open the downstream connection.

// open the downstream connection


It is now time to prepare the media object and the pipeline that will allow us to mutate the incoming feed.

We start by gaining access to the internal remote media’s stream and then select the first video track from the stream.

// get access to the internal video stream from the remote media object

var stream = remoteMedia._internal._videoMediaStream;
// get access to the video track, in this case we only have 1 so we pull the first from the array
const videoTrack = stream.getVideoTracks()[0];

We can now create our custom media pipeline. We need a trackProcessor to manage the pipeline, a trackGenerator to create our new stream after the logic has been created, and a transformer that contains our media mutation logic.

// track processor allows us to define our custom media pipeline

const trackProcessor = new MediaStreamTrackProcessor({ track: videoTrack });
// track generator is responsible for rebuilding a new stream
const trackGenerator = new MediaStreamTrackGenerator({ kind: "video" });
// transform stream holds the custom logic that can be used to mutate or read the incoming remote video frames
const transformer = new TransformStream({
  // define our transformation logic
  async transform(videoFrame, controller) {
    // in this example we will just output the video timestamp and not actually mutate the video
    // enqueue the video frame once we are done working on it

Reminder, your logic should go inside the transform function here and should be modifying the video one frame at a time.

Finally, we’ll string these objects together and output the video stream to the layout.

// string the components of the media pipeline together

// define our media stream that will come out of the generator
const streamAfter = new MediaStream([trackGenerator]);
// create a new HTML Element to host the video stream
const video = document.createElement("video");
// the the mutated stream as the video elements source
video.srcObject = streamAfter;
// add a listener to trigger the video to play
video.onloadedmetadata = function (e) {;
// add the video element to the layout

We have now completed our downstream connection handler. To recap, we instantiated our media objects, attached them to our downstream connection, pulled the internal video track off of the remote media object, and applied a transformer to generate a new video track one frame at a time.

It is now time to connect to a channel and trigger our custom remote connection logic.  We will use a button press at the starting action of this example.

// add a click handler for the start button

button.addEventListener("click", async () => {

If you have been following our blog series (WebRTC Streams before Encoder and WebRTC Remote Streams before Decoder), the next few steps will be familiar to you. We need to create our client object, create a claim for the channel we would like to join, create a client token, and register our client using the token.

// define the LiveSwitch Layout Manager helper object

let layoutManager = new fm.liveswitch.DomLayoutManager(player);
// create our client object to define our user
let client = new fm.liveswitch.Client(gateway, app);
// create a claim to join the channel of our choosing
let claims = [];
let claim = new fm.liveswitch.ChannelClaim(channelId);
// generate a token that allows us to register a client to a channel
let token = fm.liveswitch.Token.generateClientRegisterToken(
// register our client using the token
let channels = await client.register(token);
// fetch the channel object we will be using
let channel = channels[0];

The last step we need to do is add a handler that listens for when a new remote connection is opened and triggers our custom logic on that connection.

// add a handler for when a new remote connection is opened on the channel

channel.addOnRemoteUpstreamConnectionOpen(function (remoteConnectionInfo) {
  // trigger our custom downstream connection logic
  openDownstream(remoteConnectionInfo, layoutManager, channel);

We can also add another handler for the connections that were established before we joined the channel. It is a good practice to add this type of logic to your application since oftentimes you will not control the order in which users will join a session.

// handle remote connections that have already been registered on the channel before you joined

for (
  // loop through the remote connection info objects already registered
  var _i = 0, _a = channel.getRemoteUpstreamConnectionInfos();
  _i < _a.length;
) {
  // fetch the current connection info object in our loop
  var remoteConnectionInfo = _a[_i];
  // trigger the custom downstream logic for each remote connection
  openDownstream(remoteConnectionInfo, layoutManager, channel);

Congratulations! You now know how to modify any incoming remote video feed after it has already been decoded. Please note that this will increase the amount of processing being done on the receiving client’s device, and depending on the number of participants in a channel, could result in the device slowing down. As with any device-side processing, please test your logic across a number of devices and scenarios to ensure a great user experience.

If you want to see these changes live, please take a look at the CodePen.  You can also try this in your own application after signing up for our free 30-day trial.

Need assistance in architecting the perfect WebRTC application? Let our team help out! Get in touch with us today!