All Toolbar Icon Pop-ups stay open until they lose focus which means you click outside it (e.g. on the page).
It is possible to make it to automatically close after each action but this way multiple Find/Replace can be done without the need to reopen the pop-up each time.
The popup for doing the search/replace goes away, but if there is an error in the search (text is not found), THAT popup is not going away when focus is lost, it stays up for about a minute. That is the popup I meant, sorry if I was not clear. Thanks!