Hangry blog2018-05-29T11:36:35+00:00https://han-gyeol.github.ioHanGyeol OhLeook0209@gmail.comUnsupervized Image Cateogorization using Word Embedding2018-05-16T00:00:00+00:00https://han-gyeol.github.io/school/2018/05/16/Image-Categorization<p><img src="https://han-gyeol.github.io//images/2018-05-16-Image-Categorization/cover.jpeg" alt="cover" class="center-image" /></p>
<p>Social media is flooded with various categories of photos: food, travel, selfie, etc. Unless the user explicitly mentions, the category of the photos uploaded is left unknown. For the purpose of creating meaningful value from the social media photos, it might be useful to have a scalable model to classify which category each photo belongs to.</p>
<p>For this particular problem, using a supervized learning algorithm might not be scalable for several reasons:</p>
<ol>
<li>Supervized learning requires large dataset to train on, but it is hard to acquire large quantity of social media photos due to privacy issue.</li>
<li>Even if there is such dataset available publicly, it is unlikely that it also contains image category labels.</li>
<li>EVEN IF there is both data and labels available, hauman-labeled categories suffers risk of biasness. Some of the photos could sit in between two possible cateogries, who gets to decide which?</li>
</ol>
<p>Hence, using supervized learning algorithm is not a smart choice for this case, which leaves unsupervized learning algorithm pretty attractive option.</p>
<p>The unsupervized image categorization takes a couple of data processing steps:</p>
<ol>
<li>Image item labelling using Google Cloud Vision API</li>
<li>Vectorization of images using Word2Vec</li>
<li>Clustering the image vectors by K-Means algorithm</li>
</ol>
<h2 id="1-image-item-labelling-using-google-cloud-vision-api">1. Image item labelling using Google Cloud Vision API</h2>
<p>Google Cloud Vision API provides labels for items appeared in the image together with the probability and topicality values. Probability value indicates how accurate the given label is to each item, and topicality value indicates the significance of the item with respect to the rest of the items inside the image.</p>
<p><img src="https://han-gyeol.github.io//images/2018-05-16-Image-Categorization/gcv.png" alt="gcv" class="center-image" /></p>
<p>After running the Image Labelling API through each image, I have a list of item labels and their respective probability and topicality vales which are float numbers in between 0 and 1. These two values are used in the next step for weighting image vectorization using word embedding.</p>
<h2 id="2-vectorization-of-images-using-word2vec">2. Vectorization of images using Word2Vec</h2>
<p>Using the item labels for each image, the algorithm then imports pretrained Glove Word2Vec model to convert image labels into vectors. Each word vector is further weighted by the coresponding probablity and topicality values by multiplying them. Since there are generally more than one label for each image, the algorithm averages word vectors assigned for each image, producing a single centroid vector that best describe what the image is about. In the end, each image is converted to a numberical vector, so that it can be processed by Uclidean arithmetic.</p>
<h2 id="3-clustering-the-image-vectors-by-k-means-algorithm">3. Clustering the image vectors by K-Means algorithm</h2>
<p>Once all images are converted to vectors, the algorithm applies K-Means algorithm to cluster images with similar characteristic, which is bascially the type of items it contains inside.</p>
<p><img src="https://han-gyeol.github.io//images/2018-05-16-Image-Categorization/figure.png" alt="figure" class="center-image" /></p>
<p>The figure above is the visualization of the image vector space whose dimension reduced to 3, using Principle Component Analysis (PCA). The data points with the same color indicate that they belong to the same image category. In order to know what each image category represent, the programme finds the centroid vector for each image cateogry and look up the Word2Vec model to list words with the closest distance.</p>
<p><img src="https://han-gyeol.github.io//images/2018-05-16-Image-Categorization/center.png" alt="center" class="center-image" /></p>
<p>As shown by the print message, it is pretty obvious that cateogry 1 contains images related to food, while category 7 contains images related to clothings and fashion. Initially, the dataset contains uncategorized images scraped from Instagram such as these,</p>
<p><img src="https://han-gyeol.github.io//images/2018-05-16-Image-Categorization/raw.png" alt="raw" class="center-image" /></p>
<p>and classify them into arbiturally genrated categories such as food,</p>
<p><img src="https://han-gyeol.github.io//images/2018-05-16-Image-Categorization/category1.png" alt="category1" class="center-image" /></p>
<p>products,</p>
<p><img src="https://han-gyeol.github.io//images/2018-05-16-Image-Categorization/category2.png" alt="category2" class="center-image" /></p>
<p>and travel.</p>
<p><img src="https://han-gyeol.github.io//images/2018-05-16-Image-Categorization/category3.png" alt="category3" class="center-image" /></p>
<h1 id="summary">Summary</h1>
<p>This technique categorizes image dataset into a couple of arbitrary topics, based on the items spotted in each image. It first labels item labels inside each image, convert them into semantic vectors, and cluter them using K-Means algorithm. Bare in mind that the accuracy of the categorization is dependent on the accuracy on the Google Cloud Vision API and the pretrained Word2Vec model. There were few outlier images which did not give out any label when run on Cloud Vision API Labeling, which are better to be excluded out and categorized as “No Cateogry”.</p>
<p>As described above, this technique is an unsupervized learning algorithm. This means that instead of setting the categories and classifying each image into respective category, the algorithm classifies all the images first then label each image cateogry based on the common characteristics of clustered images.</p>
CS3216: Assignment 2 Application Critique2017-09-05T00:00:00+00:00https://han-gyeol.github.io/school/2017/09/05/CS3216:Assignment2-App-critique<p><img src="https://han-gyeol.github.io//images/2017-09-05-CS3216:Assignment2-App-critique/app.jpg" alt="app" class="center-image" /></p>
<p>There are several take-aways from Flowx that I can apply for my own app development:</p>
<h2 id="1-free-users-vs-primium-users-nailed-it">1. “Free Users vs Primium Users” nailed it</h2>
<p>It’s every application’s dilema that they have to charge the users somehow to generate revenue. The main problem is, app users generally have a low tolerance towards payments; they rather go for sligtly inferior free solution rather than paying for premium service. Flowx actually has Free Users and Premium Users distinction, and they provide different level of utilities to each group. What I found laudable of Flowx’s pricing policy is that they managed to separate the user experience by payment, while also satisfying both groups to a certain extent. To free users, Flowx provides basic features such as realtime rain condition and wind condition, which is just enough utilities for average users whose purpose for using the app is mainly to check the daily weather condition. On the other hand, Flowx provides over 30 types of data features to premium users, fulfilling the needs for Power users who want those meteorological data for more professional purpose. In the end, free users end up satisfied with the service because that’s exactly what they are needed, and premium users also end up satisfied because they will get the utilities they want which are not opened to Free users.</p>
<h2 id="2-too-much-ui-but-too-bad">2. Too much UI, but too bad</h2>
<p>By its nature as a meteorological app, Flowx has to display a lot of information on the screen simultaneously. They tried their best to make UI recognizable, but still some of the fonts and images are too small to recognize because everything, including the giant map, has to be crammed into small smartphone screen. Probably, this is the reason that it is relatively harder to find meteorological in mobile compared to desktop environment, because meteorological data inheritably requires a large screen size to be displayed properly. This taught be that when choosing the idea for mobile app, it is important to realize potential inherited limitations of developing in mobile environment, and if the limitations are too much, I might have to pivot to another idea instead.</p>
<h2 id="3-singapore-is-too-small-for-this">3. Singapore is too small for this</h2>
<p>One thing that the presenter pointed out was that even in the maximum zoom, Singapore is too small in map to be able to provide meaningful weather data to Singaporean users. This point is important because when an app’s target audience is in the global scale, a feature might work in one region but not so much in another due to various limitations. Flowx might have lost significant number of Singaporean users due to their map resolution limitation, and potentially other reasons that weather data in microscopic scale matters. Hence, when designing an application, the developer has to take into consideration of different conditions in different markets, and adapt to the market condition as much as possible.</p>
<h1 id="my-thoughts">My Thoughts</h1>
<p>While the idea of meteorological app is definitely cool and the UI & UX are done decently, but I believe there are some innate limitations of this application, especially in terms of generating revenue. During the presentation, the group proposed potential business ideas targeting restaurants and outdoor activities, but basic weather data such as rain activity is also available to Free users. If Flowx makes rain activity as a premium feature, they are going to lose huge number of Free users so it is not a smart move. In terms of providing advanced meteorological data to Power users, there are no significant reason that it has to come with mobility. Since desktop application provides much better experience than mobile application for the same concept, Power users are much more likely to use desktop app instead. Hence, while the app managed to attract large user pool with their cool idea and design, but I foresee limited business opportunities from their app.</p>
GSoC 2017: Wrap up2017-08-27T00:00:00+00:00https://han-gyeol.github.io/gsoc/2017/08/27/GSoC-2017:Warp-up<p><img src="https://han-gyeol.github.io//images/2017-08-27-GSoC-2017:Warp-up/gsoc.png" alt="gsoc" class="center-image" /></p>
<p>My project in this year’s Google Summer of Code was to implement <a href="https://summerofcode.withgoogle.com/projects/#4757550437236736">Micro Mode for Jitsi Meet Electron</a> project, and to work on the foundation for <a href="https://github.com/han-gyeol/jitsi-meet-spot">Jitsi Meet Spot</a> project which was newly launched by Jitsi recently.</p>
<p>GSoC Blog Posts:
<a href="https://han-gyeol.github.io/categories/#gsoc">https://han-gyeol.github.io/categories/#gsoc</a></p>
<p>Two repositories I contributed during this year’s GSoC: <br />
<a href="https://github.com/jitsi/jitsi-meet-electron/pull/12">Jitsi Meet Electron</a> <br />
<a href="https://github.com/jitsi/jitsi-meet-spot/pull/1">Jitsi Meet Spot</a> <br /></p>
<h2 id="things-i-have-learned-from-the-project">Things I have learned from the project</h2>
<ul>
<li>WebRTC P2P Connection</li>
<li>Node.js</li>
<li>Object-oriented ES6</li>
<li>Electron</li>
<li>React</li>
</ul>
<h2 id="challenges-faced">Challenges faced</h2>
<ul>
<li>Creating the Micro Mode’s remote video component faced immense amount of <a href="https://han-gyeol.github.io/gsoc/2017/07/21/GSoC-2017-Log-3/">troubles</a> due to the limitations of Electron’s inter-windows communication. In the end I used WEBRTC’s <a href="https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection">RTCPeerConnection</a> feature, which also has its own set of limitations.</li>
<li>Some of the features available in the latest version of Chromium were not available in Electron becasue the version of Chromium embeded in Electron was not the latest release. For example, I was not able to use <a href="https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/captureStream">MediaElement.captureStream()</a> feature in the Electron BrowserWindow.</li>
<li>WebRTC technology is still under development and many of its features were not supported by the browsers. For example, I had no choice but to use deprecated methods like <a href="https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/addStream">RTCPeerConnection.addStream()</a> instead of <a href="https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/addTrack">RTCPeerConnection.addTrack()</a> because addTrea() was not supported in Chrome yet.</li>
<li>The performance of the Micro Mode was not optimal as it takes up substantial amount of resources when in use, causing occasional lags when it is run together with the main Jitsi Meet Electron process. This was alleviated by firing up the Micro Mode lazily instead of on start.</li>
<li>In my implementation of Micro Mode’s toolbar, there were unnecessarily complicated layers of interfaces & abstractions I failed to resolve. These could be solved by allocating more tasks to the render process of Electron, instead of the main process. However, this is against Electron <a href="https://github.com/electron/electron/blob/master/docs/tutorial/quick-start.md#differences-between-main-process-and-renderer-process">design principle</a> which is to keep the render process as simple as possible and make it only responsible for user interface.</li>
</ul>
<h1 id="conclusion">Conclusion</h1>
<p>Google Summer of Code was my first experience of professional coding, and it certainly opened my eyes in contributing to an open source project. I have learend that frequent communication with the mentors and knowing my limits are essential to succeed in teamed programming. It was such a shame that I wasted too much time figuring out solutions for the problems that is beyond my capabilities, ended up spending less time on writing actual codes. At the end of the day, it was good to know that the amount of time I spend on the project does not necessarily reflects the amount of works I have done after the project is ended. Next time, I would definitely ask other people’s opinions more and clarify the development direction, before I start working on the actual code writing. Lots of appreciation to Saúl and Hristo for guiding me through the project, and Google for giving me such a precious opportunity.</p>
GSoC 2017: Log #52017-08-22T00:00:00+00:00https://han-gyeol.github.io/gsoc/2017/08/22/GSoC-2017:Log-5<p>I have started working on the next project, Jitsi Meet Spot. Jitsi Meet Spot is a video conference application powered by Jitsi Meet, which is suited for a physical conferecne room environment rather than a personal desktop environment.</p>
<p><a href="https://github.com/jitsi/jitsi-meet-spot/pull/1">https://github.com/jitsi/jitsi-meet-spot/pull/1</a></p>
<p>My job was to set up a HTTP server that receives command from clients, and uses <a href="https://github.com/jitsi/jitsi-meet/blob/master/doc/api.md">JitsiMeetEXternalAPI</a> to initiate the meeting, or control the conference setting such as muting the auido/video.</p>
<p>The HTTP command consists of two components: command type and arguments. Currently, it is missing the client side application, so the server can be tested by sending curl request.</p>
<p><code class="highlighter-rouge">curl --data "command=<command.type>&args=<arguments>" <targetURL></code></p>
<p>The supported commands are:</p>
<ol>
<li>join conference</li>
<li>hangup</li>
<li>toggle audio</li>
<li>toggle video</li>
<li>toggle film strip</li>
<li>toggle chat</li>
<li>toggle contact list</li>
</ol>
CS3216: Web App Development 1012017-08-07T00:00:00+00:00https://han-gyeol.github.io/school/2017/08/07/CS3216:web-app-development-101<p><img src="https://han-gyeol.github.io//images/2017-08-07-CS3216:web-app-development-101/cover.png" alt="cover" class="center-image" /></p>
<p>Web development has a very low barrier of entry apparently, looking at the internet flooded with web development tutorials. That is probably the reason that thousands of applications are released every months, which majority of people have never even heard of. The biggest trap many app developers fall into is disillusion of usefulness of their apps. Most developers try to solve problems that are not so much of troubles. They identify an inconvenience in people’s lives, develop an app that alleviate or eradicate that problem, and then find out that people find using that app more troublesome than actually solving the problem the app solves. If an application is going to be used by people, its usefulness must outweight the trouble it causes by making people use it. Almost all developers have a sense of proud for their work done, and are often misguided to think that other people will love their app as much as they do. But sadly, this is almost always not the case.</p>
<p>In order for an application to be successful, its value must come from the developers themselves, not others. All app developers must try to solve THEIR problems first, instead of solving OTHERS problems. Because only by this way, developers have a clear knowledge of ‘how troublesome’ the problem is, instead of trying to guess over other people’s opinions about the problem. This is exactly how many great softwares are born. Git was created because a Finnish developer found the existing version control solutions were a complete mess, so he decided to solve HIS problem by making a new version control solution that HE wants to use. And now, everyone uses Git.</p>
<p>In my CS3216 journey, I would like to identify problems in my life that I WANT to solve, create an application that I WANT to use. And then, I will release the app to the public, and if there are anyone who find it as useful as I do, that is one more useful app developed for others. I am really excited for the lectures and workshops which will feed me with many useful web technologies that I can use for my projects. However, I find CS3216 more of an opportunity than a learning; its resources, teammates, and great mentors attract me to this module more than anything. I would like to dive into the world of app development and test if my app development philosophy is indeed a worthy one.</p>
GSoC 2017: Phase 2 Evaluation2017-07-28T00:00:00+00:00https://han-gyeol.github.io/gsoc/2017/07/28/GSoC-2017:Phase-2-evaluation<p>By the end of Phase 2, I have completed following tasks:</p>
<ol>
<li>Create Micro window and transfer MediaElement to it.</li>
<li>Implement video transition mechanism reacting on speaker change</li>
<li>Develop basic toolbar buttons embedded in micro mode</li>
<li>Modularize and refactor the micro mode code</li>
</ol>
<p><a href="https://github.com/jitsi/jitsi-meet-electron/pull/12/commits">https://github.com/jitsi/jitsi-meet-electron/pull/12/commits</a></p>
<p>There are two modules I created for this project: ‘p2pconnection’ and ‘micromode’.</p>
<p>‘p2pconnection’ module is responsible for transferring the Jitsi-meet MediaElement from one Electron BrowserWindow to another using WebRTC technology. More details can be found in my previous post.</p>
<p>‘micromode’ module is simply a complication of all the codes I have written onto the existing codebase to make the Micro mode work. Instead of writing everything on the main.js and render.js files, I have refactored them out as a module for better readability. Currently, Jitsi-meet-electron app’s code consists of three main parts: main.js, render.js and micro.js files. ‘main.js’ file is the Main Process part of the Electron framework while ‘render.js’ and ‘micro.js’ are the Renderer Process part of the Electron, each responsible for running a BrowserWindow.</p>
<p>In each process, respective part of ‘micromode’ module is imported, initialized and disposed once the application is closed.</p>
<p><img src="https://han-gyeol.github.io//images/2017-07-28-GSoC-2017:Phase-2-evaluation/code.png" alt="code" class="center-image" /></p>
<p>As shown above, the main process simply has to require the ‘micromode’ module, and call inti, show, hide, dispose methods whenever they are necessary.</p>
<p>There are a few potential areas of improvement from the current version:</p>
<h2 id="1-add-more-features-to-micro-modes-toolbar">1. Add more features to Micro mode’s toolbar</h2>
<p>Micro mode currently has audio mute, video mute and hangup features, but there are plenty of other functionalities in the Jitsi-meet application that Micro mode can also provide, such as chat and screen sharing. Some of the features are not suitable for Micro mode, such as live stream, shared document and shared YouTube video. Screen sharing feature especially goes well with the Micro mode because the user might want to show their desktop and be able to watch the remote video at the same time.</p>
<h2 id="2-optimize-the-video-element-in-micro-mode">2. Optimize the video element in Micro mode.</h2>
<p>Micro mode occasionally has a lagging issue, either the video is a fraction of a second slower than the original video, and the transition animation is sometimes clunky. This is probably because of the WebRTC video transmission overhead or too much resources taken by Micro window. One thing I can try at the moment is to lower the video resolution in the Micro mode.</p>
<h2 id="3-switch-to-more-reliable-webrtc-technologies">3. Switch to more reliable WebRTC technologies.</h2>
<p>Currently, Micro mode’s modules several WebRTC experimental functionalities which are quite unstable and some of which are deprecated. I had no choice because there is no practical alternatives. As WebRTC get more developed, there should be follow-up maintenance works to switch to newer and more stable technologies.</p>
<h1 id="conclusion">Conclusion</h1>
<p>The more I work on this project, the more I realize the lack of IPC supports provided by Electron framework which causes spaghetti codes and runtime overhead, while giving me tremendous amount of headache during development. Nonetheless, the current version of Micro mode finally functions as its purpose, so the rest of the development would be mainly optimization works and adding more functionalities to it.</p>
GSoC 2017: Log #42017-07-26T00:00:00+00:00https://han-gyeol.github.io/gsoc/2017/07/26/GSoC-2017:Log-4<p>After resolving the largeVideo transition problem, the next step is to build a toolbar for micro window. I used Jitsi-meet’s <a href="https://github.com/jitsi/jitsi-meet/blob/master/doc/api.md">ExternalAPI</a> as the interface between the main Jitsi-meet iframe and the Micro mode.</p>
<p><img src="https://han-gyeol.github.io//images/2017-07-26-GSoC-2017:Log-4/code.png" alt="code" /></p>
<p>JitsiMeetExternalAPI class creates a Jitsi-meet iframe on initialization, and provides several methods to control its components. The most basic features I first added to the micro mode are ‘mute-audio’, ‘mute-video’ and ‘hangup’.</p>
<p><img src="https://han-gyeol.github.io//images/2017-07-26-GSoC-2017:Log-4/screen.png" alt="screen" class="center-image" /></p>
<p>Once the toolbar buttons in micro window are pressed, it sends message to the Jitsi-meet iframe through Electron’s IPC channel, and then JitsiMeetExternalAPI toggles audio and video components accordingly.</p>
<p>Whenever the micro window pops up, the main window has to send the media status to the micro window, whether user’s audio or video is muted. For now I used a ‘hacky’ way, which is to access the ‘audio-mute’ button and ‘video-mute’ button in the iframe and check whether their className includes ‘toggled’ attribute. The main window sends the video status through the IPC channel, and the micro window switches its toolbar button layout accordingly. Since accessing an iframe’s DOM element directly is not really a good practice, a better way would be using the JitsiMeetExternalAPI interface instead. It can provides some ‘get’ methods whether the video or audio inside the Jitsi-meet iframe is muted.</p>
GSoC 2017: Log #32017-07-21T00:00:00+00:00https://han-gyeol.github.io/gsoc/2017/07/21/GSoC-2017:Log-3<p>Previously, I managed to transfer the conference video from the main BrowserWindow to micro BrowserWindow using RTCPeerConnection API. However, one problem emerged was that it does not react to the dominantSpkearChanged event, which causes the transition of the largeVideo to another speaker in the Jitsi-meet application. This problem caused immense amount of trouble because it is so hard to solve without directly accessing the Jitsi-meet’s components.</p>
<p>At first, I thought of using importing Jitsi-meet’s APP object and then listen to the dominantSpeakerChanged event, but I got a feedback from the mentor that it is not a good design to directly access the components inside the iframe. So I am limited to work outside the iframe.</p>
<p>Another method recommended was to use HTMLMediaElement.captureStream() API, that returns a MediaStream object which is streaming of a real-time capture of the content being rendered in the media element. Surely, this seemed to be the ideal solution except it does not work in Electron BrowserWindow. The largeVideo DOM object does not has a “src” attribute, instead it has a “srcObject” attribute which is the MediaStream received from the current dominant speaker. Unfortunately, Chromium does not allow using captureStream() method on a video without “src” attribute, and throws the following error.</p>
<p><a href="https://chromium.googlesource.com/chromium/src/+blame/5e1a7b0dd27f9eaed596106ee2726cab03df07c9/third_party/WebKit/Source/modules/mediacapturefromelement/HTMLMediaElementCapture.cpp#24">Uncaught NotSupportedError: Failed to execute ‘captureStream’ on ‘HTMLMediaElement’: The media element must have a source.</a></p>
<p>This has been resolved in the Chromium version 59.x.x.x, but even the latest version of Electron, v1.7.5 beta uses Chromium version 58.x.x.x. So there is no legal way to use HTMLMediaElement.captureStream() on the Jitsi-meet’s largeVideo in Electron environment.</p>
<p>One work around I tried was to make a canvas copy of the largeVideo, and apply <a href="https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream">HTMLCanvasElement.captureStream()</a>.</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="cm">/**
* Create a copy of HTMLvideo in HTMLcanvas for
* Get reference of the video, width and height as parameters
* Return canvas object
*/</span>
<span class="kd">function</span> <span class="nx">copyVideo</span><span class="p">(</span><span class="nx">video</span><span class="p">,</span> <span class="nx">width</span><span class="p">,</span> <span class="nx">height</span><span class="p">,</span> <span class="nx">frameRate</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">canvas</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="s1">'canvas'</span><span class="p">);</span>
<span class="nx">canvas</span><span class="p">.</span><span class="nx">width</span> <span class="o">=</span> <span class="nx">width</span><span class="p">;</span>
<span class="nx">canvas</span><span class="p">.</span><span class="nx">height</span> <span class="o">=</span> <span class="nx">height</span><span class="p">;</span>
<span class="kd">let</span> <span class="nx">ctx</span> <span class="o">=</span> <span class="nx">canvas</span><span class="p">.</span><span class="nx">getContext</span><span class="p">(</span><span class="s1">'2d'</span><span class="p">);</span>
<span class="nx">setInterval</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">ctx</span><span class="p">.</span><span class="nx">drawImage</span><span class="p">(</span><span class="nx">video</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">canvas</span><span class="p">.</span><span class="nx">width</span><span class="p">,</span> <span class="nx">canvas</span><span class="p">.</span><span class="nx">height</span><span class="p">);</span>
<span class="p">},</span> <span class="nx">frameRate</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">canvas</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>This approach worked and I was able to capture the Jitsi-meet’s largeVideo onto a canvas element, which reacts to the video transition event properly.</p>
<p>I really hoped this canvas approached solve the video transition problem for once and for all, but it did not. Once the window get minimized, the canvas stop capturing the video, probably because for most of the cases there is no point of capturing visual data when the window itself is not visible.</p>
<p>After some desperate researching and all alternatives failed, I posted a question in StackOverflow and got an unexpected solution for this long dragged problem.</p>
<p><a href="https://stackoverflow.com/questions/45156701/html-canvas-drawimage-when-window-minimized">https://stackoverflow.com/questions/45156701/html-canvas-drawimage-when-window-minimized</a></p>
<p>It turned out that HTMLVideoElement.ondurationchange event can detect the change in srcObject of a video element. In this way, I don’t have to access the Jitsi-meet iframe’s App component and still be able to react to the dominantSpeakerChanged event. After discovering this, the rest of the process was very easy; whenever ondurationchange event is emitted, the main window remove the existing MediaStream object from the p2pChannel and attach the new MediaStream object.</p>
<p>I have been struggling so long to solve this problem yet the answer was somewhat disappointingly simple. One thing I realized was that WebRTC is a fairly new technology and a lot of its parts are under development. Even the <a href="https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/addStream">addStream()</a> method I used to transfer the MediaStream object is deprecated, and was recommended to use <a href="https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/addTrack">addTrack()</a> method instead. However, addTrack() method is not supported in Chrome, so I was forced to use a deprecated method. This is not the only case of using experimental features in WebRTC, so I foresee lots of upcoming maintenance works as some of the features become unsupported in near future.</p>
Game AI: Snake Game2017-07-08T00:00:00+00:00https://han-gyeol.github.io/fun/2017/07/08/Game-AI:Snake-Game<p><img src="https://han-gyeol.github.io//images/2017-07-08-Game-AI:Snake-Game/snake.gif" alt="snake" class="center-image" /></p>
<p style="text-align: center;"><a href="https://github.com/leook0209/Genetic-Algorithm-Snake">https://github.com/leook0209/Genetic-Algorithm-Snake</a></p>
<p>Snake game is a simple game in which the player moves the head of the snake up, down, right or left to eat a randomly generated food. The snake grows its size by one every time it eats the food, and the snake dies once it hits any part of its body. This project is about training an utility-based snake game agent using a genetic algorithm with a number of heuristics.</p>
<h1 id="fitness">Fitness</h1>
<p>Goal(fitness) of each game is to have the snake’s length as long as possible, while taking as minimum as possible turns to finish the game. Fitness is calculated by following way:</p>
<center> fitness = Length of Snake- α * ( Number of Turns taken) </center>
<center> α: weight for Number of Turns Taken </center>
<p>The reason for minimizing the number of turns is that snake game has a very easy strategy to beat, which is to circle the snake around the edge and eat the food at the inner side of the field in a safe manner. Hence, α should be set with a reasonable number in order to prevent the agent to take an easy way out.</p>
<p>At each turn, the agent calculates a heuristic value for moving each direction: up, down, left and right. If a direction leads to the snake’s death, the heuristic value is NEGATIVE INFINITY. Even if a position next to the head contains the food, the snake might not decide to take it if it leads to a less desirable future state (e.g. creating a dead end). To prevent the Snake from taking the same motion over and over again, it is designed to be more attracted towards the food with time.</p>
<h1 id="heuristics">Heuristics</h1>
<p>There are 6 heuristics the agent uses to calculate the fitness:</p>
<ol>
<li>Manhattan distance between the Snake’s head and Food</li>
<li>The position of the Snake’s head from the center of the field</li>
<li>Squareness of the Snake</li>
<li>Compactness of the Snake</li>
<li>Connectivity of the field</li>
<li>Dead End Indicator</li>
<li>First two heuristics are quite intuitive to understand while next four heuristics are not. Those are heuristics concepts I created for this agent.</li>
</ol>
<h2 id="squareness">Squareness</h2>
<p>Squareness is an indicator of how the Snake’s body is orientated in a square/rectangular manager.</p>
<center> O – Empty Space, S – Snake, X – Blank Space, H – Snake’s Head
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">H</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span></code></pre></figure>
</center>
<p>The above example’s snake is oriented in a perfect rectangular manner. In this case, the squareness value is 0.</p>
<center>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">X</span> <span class="nx">X</span> <span class="nx">X</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">X</span> <span class="nx">X</span> <span class="nx">X</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">X</span> <span class="nx">X</span> <span class="nx">H</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span></code></pre></figure>
</center>
<p>The squareness value is the number of blank spaces that is within the Snake’s square boundaries but not filled by the snake. The square boundaries refers to the rectangular space taken up by the leftest, rightest, upper most, and lower most part of the snake. For the above case, the squareness value is the number of Xs, which is 8.</p>
<h2 id="compactness">Compactness</h2>
<p>Compactness is an indicator of how compactly the Snake’s body is oriented. It is the number of cases where one body part of the Snake is placed next to another body part of the Snake, without double counting.</p>
<center> O – Empty Space, S – Snake, H – Snake’s Head
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">H</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span></code></pre></figure>
</center>
<p>For the above case, the compactness of the Snake is 10.</p>
<h2 id="connectivity">Connectivity</h2>
<p>Connectivity is an indicator of how connected each part of the field is, and whether the Snake is separating one part of the field from another.</p>
<center> O – Empty Space, S – Snake, H – Snake’s Head, X – Space Chosen by Agent
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">H</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">X</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span></code></pre></figure>
</center>
<p>At each turn, the agent pick a random empty space in the field, and count how many spaces are disconnected from that space as they are blocked by the Snake’s body. For above case, the connectivity is 148.</p>
<h2 id="dead-end-indicator">Dead End indicator</h2>
<p>Dead End Indicator represents how many spaces are unreachable by the snake based on the current orientation.</p>
<center> O – Empty Space, S – Snake, H – Snake’s Head
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">H</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span>
<span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">S</span> <span class="nx">S</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span> <span class="nx">O</span></code></pre></figure>
</center>
<p>Dead End indicator is calculated in a similar manner to the Connectivity, except that instead of choosing a random empty space, it checks connectivity from the Snake’s head. For above case, the left side of the field is unreachable from the Snake’s head, hence the Dead End Indicator value is 134.</p>
<p>After a few rounds of training, the genetic algorithm shows a trend to maximize the Compactness and minimize the Distance from Food, Squareness, Connectivity and Dead End, while it does not really care about the Distance from the center of the field.</p>
<h1 id="genetic-algorithm">Genetic Algorithm</h1>
<p>The genetic algorithm for the Snake Game agent has a population size of 500, mutation rate of 0.05, survival rate of 0.5. For each weight sets, the game is played for 3 times and taken the arithmetic average, in order to minimize the effect of randomly generated food positions.</p>
<h2 id="crossover">Crossover</h2>
<p>At each generation, the population is sorted based on their fitness value, and two parents are chosen from the 50% of the surviving population, while a Snake with higher fitness having a proportionally higher chance of being chosen. A child is created by taking the weighted average of each heuristic weight of the parents’.</p>
<h2 id="mutation">Mutation</h2>
<p>Each heuristic weight of the Snake in the population has 5% chance of being mutated by ±0.2.</p>
<p>The biggest learning point was that it is a bad idea to create a computation intensive program with Web (JavaScript). Due to the performance limitation of the Chrome browser, it took so much time to train the agent with the Genetic Algorithm of population 500. Even after the training, the elite Snake still could not finish the game. The most difficult part is that the food can be generated at an unreachable position. Hence, it is wise to minimize the number of holes generated by the Snake’s body (Connectivity) but then it takes too many steps to clear the game.</p>
GSoC 2017: Phase 1 Evaluation2017-06-29T00:00:00+00:00https://han-gyeol.github.io/gsoc/2017/06/29/GSoC-2017:Phase-1-evaluation<p>In the first month of the project, I have attempted following tasks:</p>
<ol>
<li>Capture the large video embedded inside the Jitsi-meet’s iframe in the main renderer BrowserWindow</li>
<li>Transmit to the micro mode’s BrowserWindow</li>
<li>Display on a HTML video element</li>
</ol>
<h1 id="1-capture-the-html-video-element-inside-iframe">1. Capture the HTML Video Element inside iframe</h1>
<p><img src="https://han-gyeol.github.io//images/2017-06-29-GSoC-2017:Phase-1-evaluation/source.png" alt="source" class="center-image" /></p>
<p>The Jitsi-meet’s largeVideo can be extracted directly from its iframe, using</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="nx">iframe</span><span class="p">.</span><span class="nx">contentWindow</span><span class="p">.</span><span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="s1">'largeVideo'</span><span class="p">);</span></code></pre></figure>
<p>I can subsequently retrieve the source MediaStream from the video’s srcObject attribute. However, if I simply display that MediaStream on the micro mode’s window, it does not react when the dominant speaker changes in the Jitsi-meet application, because the original HTML video’s srcObject attributes switches to another MediaStream.</p>
<p>There are two possible options to implement video transition in Micro Mode:</p>
<ol>
<li>A ‘hacky’ way. Import Jitsi-meet’s APP object and listen to the dominantSpeakerChanged event. Once the speaker changes, re-extract the largeVideo from the iframe.</li>
<li>Capture the largeVideo displayed on the main BrowserWindow, convert it to MediaStream and send to the Micro Mode’s window. When the speaker changes, it automatically captures the video transition.</li>
</ol>
<p>So far, I have attempted the second approach, but it has several problems. The existing version of HTMLMediaElement.captureStream() method does not work because the largeVideo extracted from the iframe lacks ‘currentSrc’ attribute, hence keep throwing “The media element must have a source.” error. I need to find an alternative of HTMLMediaElement.captureStream() that captures a HTML video element real time, and produces a MediaStream object.</p>
<p>One approach I tried was using the HTML canvas to take a snapshot of each frame of the largeVideo and render it like a video.</p>
<h1 id="2-transmit-the-video-to-micro-window-using-webrtc">2. Transmit the Video to Micro Window using WebRTC</h1>
<p>The inherited difficulty of this task is that each Electron BrowserWindow is an independent Chromium page. There is virtually no direct way to transfer media data from one window to another. After a long research, it was concluded that using webkitRTCPeerConnection to set up a MediaStream peer connection between the main window and the micro window is the most feasible approach.</p>
<p>Reference: <a href="https://www.tutorialspoint.com/webrtc/webrtc_video_demo.htm">https://www.tutorialspoint.com/webrtc/webrtc_video_demo.htm</a></p>
<p>The details of how peer connection is implemented are shown in the <a href="https://han-gyeol.github.io/blog//2017/05/17/GSoC-2017-Log-1/">GSoC 2017: log #1</a>.</p>
<p>Since the main audio is played in background after the Jitsi-meet window is minimized, there is no need to transmit the audio to the micro window.</p>
<p>One major concern is the performance issue. Running a background peer connection between the main window and the micro window might cause a performance drop of the Jitsi-meet conference.</p>
<h1 id="3-retrieve-the-video-and-display-on-micro-window">3. Retrieve the Video and Display on Micro Window</h1>
<p>After the MediaStream of the largeVideo is received by the micro window’s side, it can be simply displayed by setting the srcObject attribute of the HTML video. Then, the micro mode’s window is positioned on the top right corner of the screen, with the frameless option and always-on-top options enabled. The end product looks like this…</p>
<p><img src="https://han-gyeol.github.io//images/2017-06-29-GSoC-2017:Phase-1-evaluation/evaluation.png" alt="evaluation" /></p>
<p>However, several problems emerged after testing minimization of the main window.</p>
<p>Using a HTML canvas to capture the large video works only when both windows(main and micro) are active and visible. When I tried minimizing the main window, the video in micro window just freezes. I am guessing that HTML canvas’s snapshot method works only when the target video is active(not minimized). I need to search if it is possible to play the target video in background even after the window is minimized.</p>
<p>Furthermore, when I tested sending a local video(mp4) from the main window to the micro window, videos in both windows ran smoothly. However, once I minimize the main window, frame rate of the micro window’s video immediately starts dropping. It is still playing, but the frame rate drops quite seriously that it does not look like a video anymore. I am going to research how Chromium browser handles a background Media, and if there is a way to activate a target MediaElement in background.</p>
<h1 id="conclusion">Conclusion</h1>
<p>Although the video transmission appears to be working, there are numerous internal problems I have to resolve before I move on to the next step. A lot of time is consumed for researching possible methods to implement the features, and only a fraction of time is used for the actual code writing. I used to only look up StackOverFlow for my programming problems, but for this project I had to read on many official API documentations, issue & bug trackers, and discussion threads. This is because the problems I used to solve were a kind that had one simple solution which worked cleanly. But for the Jitsi-meet-electron project, there are many different approaches I can take to solve the same problem, and in worst case scenario, the problem is actually unsolvable / not supported.</p>
GSoC 2017: Log #22017-06-28T00:00:00+00:00https://han-gyeol.github.io/gsoc/2017/06/28/GSoC-2017:Log-2<p>Previously, I have set up a RTCPeerConnection between the main jitis-meet window and the micro window for the video transmission.</p>
<p>However, it hard coded a lot of stuffs and honestly speaking, the code design was a total mess. There were both main window side’s code and micro window side’s code in one page, so it was vulnerable to unintended global variables usage and namespace conflicts. I decided to refactor the code into OOP-style one, so that it can be used in more general case. It is sort of a wrapper for the webkitRTCPeerConnection, suited for the inter-Electron BrowserWindow communication which does not require a STUN server for signalling.</p>
<h1 id="api">API</h1>
<h3 id="class-windowpeerconnection">Class: WindowPeerConnection</h3>
<h4 id="parameter">Parameter</h4>
<ul>
<li><strong>windowName</strong>: the BrowserWindow’s name in main process</li>
</ul>
<p>The WindowPeeerConnection wraps around webkitRTCPeerConnection class, which on construction in a renderer process, sets up a IPC listeners that receive message and data from other windows. All the message are relayed through the main process, and data are serialized before transmitted.</p>
<h4 id="method">Method</h4>
<ul>
<li><strong>attachStream ( streamToSend )</strong>: attaches a MediaStream object on the peer connection channel, which can be transmitted by sendStream method. Currently, it can only attach one stream per connection.</li>
<li><strong>removeStream ( )</strong>: removes the MediaStream attached to this peer connection channel.</li>
<li><strong>sendStream ( receiverName )</strong>: sends the attached MediaStream object to the target window specified the string ‘receiverName’. receiverName refers to the name of the variable assigned to the target BrowserWindow in the Electron’s main process.</li>
</ul>
<h4 id="event-listener">Event Listener</h4>
<ul>
<li><strong>onReceivedStream ( function(receivedStream) )</strong>: triggered when the WindowPeerConnection receives a MediaStream from a remote window.</li>
</ul>
<h3 id="main-process-data-relay-channel">Main Process: Data Relay Channel</h3>
<p>A Javascript object that is called in the Electron’s main process. Since Electron does not support direct IPC between BrowserWindows, all messages and data must be relayed through the main process, using ipcMain. Each BrowserWindow is wrapped as a client in the data Channel.</p>
<h4 id="function">Function</h4>
<ul>
<li>
<p><strong>addClient ( client )</strong>: adds a client on the data relay channel. A client is a Javascript object with the following format: client = { window: browerWindowObject, name: “BrowserWindowName” }</p>
</li>
<li>
<p><strong>removeClient ( clientName )</strong>: removes a client with the given window name from the data relay channel.</p>
</li>
<li>
<p><strong>initChannel ( )</strong>: initiates the data relay channel with the current array of clients. Clients can be furthered added after the channel is initiated.</p>
</li>
<li>
<p><strong>dispose ( )</strong>: disposes the data relay channel.</p>
</li>
</ul>
<h2 id="example">Example</h2>
<h4 id="main-process">Main process</h4>
<p><img src="https://han-gyeol.github.io//images/2017-06-28-GSoC-2017:Log-2/main.png" alt="main" /></p>
<h4 id="sender-process">Sender process</h4>
<p><img src="https://han-gyeol.github.io//images/2017-06-28-GSoC-2017:Log-2/window1.png" alt="Sender" /></p>
<h4 id="receiver-process">Receiver process</h4>
<p><img src="https://han-gyeol.github.io//images/2017-06-28-GSoC-2017:Log-2/window2.png" alt="Receiver" /></p>
<p>Currently, it only supports transmission of MediaStream, but it could be extended to general Data transmission later on. I might be going to publish this code in npm, since I could find anything similar.</p>
GSoC 2017: Log #12017-06-22T00:00:00+00:00https://han-gyeol.github.io/gsoc/2017/06/22/GSoC-2017:Log-1<p>First of all, I need to make sure that the large video in the Main window’s Jitsi-meet iframe also appears in the Micro window. If I can’t even access the video elements in the Jitsi-meet iframe, I can’t proceed to anywhere.</p>
<p>There were three approaches I attempted:</p>
<ol>
<li>Import the video element directly from the render process, using ‘module require’.</li>
<li>Use <a href="https://developer.mozilla.org/en-US/docs/Web/API/URL/createObjectURL">createObjectURL</a> to generate URL of the video element and pass it to Micro mode’s window.</li>
<li>Instead of creating a separate window for Micro mode, change the layout of main window when the window is minimized.</li>
</ol>
<p>After trying all three, I realize that none of them actually works. The most important problem I missed out was that each Electron BrowserWindow is an separate, independent process, and Electron does not provides a Inter-Process-Communication(IPC) feature for media elements.</p>
<p>The reasons each approach failed are:</p>
<p>Node’s ‘module require’ does not support transferring of a HTMLMediaElement to another page. It became ‘undefined’ every time I attempted.
Similar to the ‘require’ approach, the scope of URL create is limited within the same page. When I tried to receive the media element from the URL, I only got 404 response.
This approach only works partially. I can restructure the layout every time the Micro mode is called, but that means the Normal mode and the Micro mode shares the same window. So when the user minimizes the (only) window, the Micro mode’s window gets minimized as well.</p>
<p>In the end, I ended up using webRTC’s <a href="https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection">RTCPeerConnection</a> to set up a peer-to-peer connection between the main window and the micro window. By right, I was supposed to use a <a href="https://en.wikipedia.org/wiki/STUN">STUN/TURN server</a> to do signalling between the main window and the micro window. But since they are both under the Electron’s main process, I used Electron’s IPC feature instead to pass necessary information for signalling.</p>
<p>After creating html and js files for the Micro mode, I added a module for the peer connection between the main window and the micro window.</p>
<p>The WebRTC connection is established in following order:</p>
<table>
<thead>
<tr>
<th>Main Window</th>
<th>Micro Window</th>
</tr>
</thead>
<tbody>
<tr>
<td>1. Sets up RTCPeerConnection</td>
<td></td>
</tr>
<tr>
<td></td>
<td>2. Sets up RTCPeerConnection</td>
</tr>
<tr>
<td>3. Attaches jitsi video stream to peer connection</td>
<td></td>
</tr>
<tr>
<td>4. Creates and sends connection offer</td>
<td></td>
</tr>
<tr>
<td></td>
<td>5. Receives the offer</td>
</tr>
<tr>
<td></td>
<td>6. Creates and sends answer</td>
</tr>
<tr>
<td>7. Receives the answer</td>
<td></td>
</tr>
<tr>
<td>8. Sends ICE candidate</td>
<td></td>
</tr>
<tr>
<td></td>
<td>9. Receives and adds ICE candidate</td>
</tr>
<tr>
<td></td>
<td>10. Receives the media stream</td>
</tr>
</tbody>
</table>
<p>And then… Voila!
<img src="https://han-gyeol.github.io//images/2017-06-22-GSoC-2017:Log-1/rtc.png" alt="rtc" /></p>
GSoC 2017: Log #02017-05-17T00:00:00+00:00https://han-gyeol.github.io/gsoc/2017/05/17/GSoC-2017:Log-0<p>This summer is for code. My proposal has been accepted to the <a href="https://developers.google.com/open-source/gsoc/">Google Summer of Code(GSoC) 2017</a>, so I would be working under <a href="https://jitsi.org/">Jitsi</a> for next few months. I never thought I would be accepted as this was my first attempt to GSoC, but hell yeah.</p>
<p><img src="https://han-gyeol.github.io//images/2017-05-17-GSoC-2017:Log-0/gsoc.png" alt="gsoc" /></p>
<h3 id="project-implementing-micro-mode-for-jitsi-meet-electron">Project: Implementing ‘Micro Mode’ for Jitsi-Meet-Electron</h3>
<p>URL: <a href="https://summerofcode.withgoogle.com/projects/#4757550437236736">https://summerofcode.withgoogle.com/projects/#4757550437236736</a></p>
<p>This is a simple feature for Jitsi-Meet’s desktop app, which pops up a small video and set of GUIs when the user minimizes the main window. It is a similar one to the one in Skype, hope I can make it cooler than that.</p>
<p>So excited to start working on the project, and many thanks in advance to the mentors <a href="https://github.com/hristoterezov">Hristo Terezov</a> and <a href="https://github.com/saghul">Saúl Ibarra Corretgé</a> who will be guiding(tolerating) me for the period of time. Hope I won’t fall behind.</p>
Beginning of All Small Things2017-04-02T00:00:00+00:00https://han-gyeol.github.io/2017/04/02/beginning-of-all-small-things<p><img src="https://han-gyeol.github.io//images/2017-04-02-beginning-of-all-small-things/beginning.jpg" alt="beginning" /></p>
<p>I have lots of daily useless thoughts. I guess I can use this blog to pour them out, so I can empty my brain for more productive things. Should I write this blog in English or Korean? Well, apparently I am already writing in English…</p>