Run/scrape web site with Extension?

Add-ons Development
#1

Is it possible via an extension to automatically operate a web site? I currently use a keyboard macro program to run Firefox, log in, navigate to specific pages and download a file. This process is erratic since the response time from the web site can vary widely and if the delay windows are not enough then the process becomes out of sync. The http TCP objects in .NET aren’t helpful since the site is HTTPS and it is not possible to see what is being transmitted/received from the site to emulate the responses.

Bascially what I need is to have an extension go to a URL, Wait until it loads, Fill login and password fields, Post, verify login via returned content, then navigate to URL links in the site to initiate a download, Close Firefox.

Is this something an extension can even do? Is it the right technology?

Any suggestions would be greatly appreciated.

#2

A browser extension can probably do what you want. You should also consider some alternatives like JSDOM and PhantomJS.

#3

the site is HTTPS and it is not possible to see what is being transmitted/received

You can use the network tab of any decent browsers dev tools to inspect HTTPS traffic.

WebExtensions can do most of what you describe (but AFAIK can’t close the browser), but are certainly not the automation tool you actually want.
If the website in question is complex, something like JSDOM, PhantomJS or a headless browser controlled by selenium may be necessary, but for most sites, simple HTTPS requests and some reverse-engineering of the cookies and headers they use should give the most reliable result.