GSoC 2017: Log #3

Previously, I managed to transfer the conference video from the main BrowserWindow to micro BrowserWindow using RTCPeerConnection API. However, one problem emerged was that it does not react to the dominantSpkearChanged event, which causes the transition of the largeVideo to another speaker in the Jitsi-meet application. This problem caused immense amount of trouble because it is so hard to solve without directly accessing the Jitsi-meet’s components.

At first, I thought of using importing Jitsi-meet’s APP object and then listen to the dominantSpeakerChanged event, but I got a feedback from the mentor that it is not a good design to directly access the components inside the iframe. So I am limited to work outside the iframe.

Another method recommended was to use HTMLMediaElement.captureStream() API, that returns a MediaStream object which is streaming of a real-time capture of the content being rendered in the media element. Surely, this seemed to be the ideal solution except it does not work in Electron BrowserWindow. The largeVideo DOM object does not has a “src” attribute, instead it has a “srcObject” attribute which is the MediaStream received from the current dominant speaker. Unfortunately, Chromium does not allow using captureStream() method on a video without “src” attribute, and throws the following error.

Uncaught NotSupportedError: Failed to execute ‘captureStream’ on ‘HTMLMediaElement’: The media element must have a source.

This has been resolved in the Chromium version 59.x.x.x, but even the latest version of Electron, v1.7.5 beta uses Chromium version 58.x.x.x. So there is no legal way to use HTMLMediaElement.captureStream() on the Jitsi-meet’s largeVideo in Electron environment.

One work around I tried was to make a canvas copy of the largeVideo, and apply HTMLCanvasElement.captureStream().

/**
 * Create a copy of HTMLvideo in HTMLcanvas for
 * Get reference of the video, width and height as parameters
 * Return canvas object
 */
function copyVideo(video, width, height, frameRate) {
  let canvas = document.createElement('canvas');
    canvas.width = width;
    canvas.height = height;
    let ctx = canvas.getContext('2d');
    setInterval(function() {
      ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
    }, frameRate);
    return canvas;
}

This approach worked and I was able to capture the Jitsi-meet’s largeVideo onto a canvas element, which reacts to the video transition event properly.

I really hoped this canvas approached solve the video transition problem for once and for all, but it did not. Once the window get minimized, the canvas stop capturing the video, probably because for most of the cases there is no point of capturing visual data when the window itself is not visible.

After some desperate researching and all alternatives failed, I posted a question in StackOverflow and got an unexpected solution for this long dragged problem.

https://stackoverflow.com/questions/45156701/html-canvas-drawimage-when-window-minimized

It turned out that HTMLVideoElement.ondurationchange event can detect the change in srcObject of a video element. In this way, I don’t have to access the Jitsi-meet iframe’s App component and still be able to react to the dominantSpeakerChanged event. After discovering this, the rest of the process was very easy; whenever ondurationchange event is emitted, the main window remove the existing MediaStream object from the p2pChannel and attach the new MediaStream object.

I have been struggling so long to solve this problem yet the answer was somewhat disappointingly simple. One thing I realized was that WebRTC is a fairly new technology and a lot of its parts are under development. Even the addStream() method I used to transfer the MediaStream object is deprecated, and was recommended to use addTrack() method instead. However, addTrack() method is not supported in Chrome, so I was forced to use a deprecated method. This is not the only case of using experimental features in WebRTC, so I foresee lots of upcoming maintenance works as some of the features become unsupported in near future.