Hi,
I develop Ultrawidify, which is an extension that crops letterboxed videos to fit a 21:9 monitor properly. One of the features is automatic aspect ratio detection.
The way automatic autodetection works is by taking the video element, drawing current frame to canvas every X seconds with ctx.drawImage() and then getting pixels from the canvas through the magic of ctx.getImageData().
Main function looks roughly like this — I’ve omitted some less critical parts:
async main() {
while (cond) { // yes I know. setInterval exists.
checkFrame();
await sleep(interval);
}
}
And the checkFrame() boils down to this:
async checkFrame() {
this.context.drawImage(this.video, 0, 0, this.canvas.width, this.canvas.height);
const imageData = this.context.getImageData(0, 0, this.canvas.width, this.canvas.height).data; // <----- problem child.
// do stuff with imageData, lots of stuff
// check results
// return nothing
}
As far as I can tell, that’s pretty much by the book. My code has no memory leaks and there’s no way to improve my interactions with canvas at all. Reusing existing references is not an option because ctx.getImageData() will always return a new one. After checkFrame() finishes executing, imageDatashould get garbage-collected eventually because unused reference.
And that’s what usually tends to happen. If I open a youtube video, ‘memory’ tab in devtools will peg memory usage for the page at 70-120 MB, which is reasonable. After a while though, the memory usage is going to start to rise (personal record: to the tune of 20+ GB and no I’m not kidding).
Today, I’ve only managed to get it up to 1 gig, but that’s still way too much: https://imgur.com/btcUlpO
If you take a look at the ‘dominator’ view, you start noticing funny stuff: there’s tons of ArrayBuffer objects 921664B big:
Which is coincidentally roughly how big I expect imageData to be (4 bytes per pixel × 640 pixels wide × 320 pixels tall = pretty much this).
Going to ‘about:memory’ and clicking ‘GC’ button will bring the number back from multiple gigabytes to what it should be — notice drops in fourth and last snapshot:
Autodetection is the kind of feature I don’t want to go back on, and it’s mildly important that you run it frequently enough in case video keeps changing aspect ratio. Are there any lesser-known workarounds that would help me curb the RAM usage due to shitty garbage collection?
Things I’ve tried so far
- Googling. No results.
Things that I’m looking at
- Web workers.
I’ve found this bit about transferable objects. If I understand this right, sending imageData to a worker like suggested here and killing the worker once it’s done processing would serve as a kind of forced garbage collection — or am I wrong on this one?
Any luck on solving the overall problem though?