Can sound out of Firefox be globally filtered by an Add-On?

Add-ons Development
#1

Is it possible to write an add-on that grabs the audio stream as it emerges from Firefox and manipulate it before sending it on its way?

If so I would like to find a programmer hacker that knows how to do that, and simply prove that to get paid to do it, to create an add-on that merely connects to a single VST host and lets it play havoc any way a creative sound engineer can provide before passing it on to whatever bit bucket Firefox sound usually goes into.

You can trade payment for equity. For 100% equity, no payment, it’s all yours, child, I just get the joy of helping make it and using it to solve another problem I’m brickwalled against. I think there is money potential enough (it will become a ubiquitous if not native capability) by itself that somebody who know how should jump on it. If it fails to spark that inevetibility feeling in anyone I will shell out some of my hard saved to get it. I can be a useful mentor as well to help you learn the lay of the land.

Even if there is not a current spot in the add-on API for processing sound I can’t imagine the hack to provide it would prove to even be a difficult task for someone with the right skill set. I’m willing to even consider a FF executable patch with legs. Then one needs to be able to, from the add-on’s JavaScript, invoke an external interface to exchange an audio stream to that, a VST host with a GUI running on the user’s machine. There are many superlative candidates for that host function. The add-on needs provide nothing for management. It’s only user interface is the ability to name a VST host that’s on one’s machine. Then wait for the contributions to roll in. :slight_smile:

Even if you just carry the task to where any JavaScript programmer can use your new API as a filter I might then be able to find someone who can do the rest regarding hooking up with a running app to hand it samples and return the results to the Firefox stream.

1 Like
#2

I dont know of any APIs in Firefox that allow this. But when they land recording browser sound in webrtc (in the works) you can probably use that. But I don’t know if you can manipulate that sound before it goes out.

You have a very specific need so maybe you won’t find someone to do this for you. You should take a stab at it yourself, that would be awesome.

I would wait to hear from others before going with the method I’m going to mention below.

platform method

With platform methods yes you can do this. Be prepared to read docs though, some of which are poor docs, and be prepared to search code examples in non-js.

On Windows I was playing around with COM via ctypes to route sound-in to a mic straight to the speakers-out. I used the DirectShow API. With some research on this API you can easily change the source from sound-in-from-mic to sound-out-from-app - and you can manipulate it -

https://github.com/Noitidart/ostypes_playground/commits/audio-capture

I had followed these tutorials:

http://www.codeproject.com/Articles/31022/A-simple-console-audio-input-device-reader

http://www.codeproject.com/Articles/31472/Reading-audio-video-devices-and-codecs-in-DirectSh

http://www.codeproject.com/Articles/31788/Audio-Capture-with-DirectShow-Part

#3

Thank you so much for the advice. That’s more than I have the chops to attempt by a long shot but I know enough to know how useful the information you’ve given will be to anybody who wants to grab the ring. Yourself, perchance? You seem to have a head start. :slight_smile:

#4

P.S. For me your handle noitidart presents a real visual conundrum. I seem to have an autistic type reaction to it where I have to study it a bit to figure it out.

#5

I couldn’t lead the effort. However I will gladly help you or whoever works on its. Just post here.

Also, before you go at it with ctypes, let’s wait to see if anyone else posts any other way. It’s not clear if ctypes will be deprecated in the future.

#6

I think most of the audio playback stuff happens with code defined in https://dxr.mozilla.org/mozilla-central/source/dom/media (except for nsISound, probably). I don’t know how you would access it, though.

EDIT: I don’t think Firefox uses the system’s gstreamer, so you couldn’t just give that a different default sink to re-route the output.

#7

If it’s ok that it only supports and elements, and not plugins (no idea about DRM protected audio), then you could do this:

  • Inject a content script into every tab
  • Have a MutationObserver listen for and elements
  • Use the WebAudio API to manipulate the sounds

A small (untested) example content script:

function handleElement(element) {
    const context = new AudioContext;

    // grab audio
    const source = context.createMediaElementSource(element);

    // do fancy stuff
    const gain = context.createGain();
    gain.gain.value = 2;

    source.connect(gain);

    // output
    gain.connect(context.destination);
}

new MutationObserver(mutations => forEach(mutations, ({ addedNodes, }) => forEach(addedNodes, element => {
    const tag = element.tagName;
    if (tag === 'AUDIO' || tag === 'VIDEO') {
        handleElement(element);
    } else if(tag && element.querySelectorAll) {
        Array.prototype.forEach.call(element.querySelectorAll('audio, video'), handleElement);
    }
}))).observe(document, { subtree: true, childList: true, });
1 Like
#8

Wow! This is super cool! Thanks @NilkasG!