Migrate from BrowserUp Proxy to another HAR solution

[basil]

HAR recording currently uses the BrowserUp Proxy library, which is archived:

ANNOUNCEMENT:

-- The BrowserUp Proxy is now deprecated. --

We have moved our business over to the mitmproxy. It is available here

The reasons:

After multiple tries, we could not make the current architecture support websockets and http/2. without a massive rewrite. HTTP/3 will complicate things even further.

mitm proxy is great! The upstream maintainers are awesome. We have, and hope to continue to make code contributions.

• It has HTTP/2 Support
• It has Websocket Support

Not only is it end-of-life, but it is explicitly documented not to support WebSocket, which we now rely on via WebDriver BiDi. While it does not fail outright, it is flaky. Local testing has shown that flakiness disappears when disabling HAR recording.

---

The recommended replacement is mitmproxy, which is not a Java library but a Python application. There is an Ubuntu package for it as well as a Docker image. For the purposes of this repository, which uses an all-in-one Docker image containing Firefox and everything else, the Ubuntu package would make sense. For CloudBees, which uses Docker Compose and multiple Docker containers, the Docker image would make more sense. So the implementation would ideally support both, for example using an environment variable to specify the proxy URL for the Docker use case, or otherwise starting the process itself.

[basil]

@jtnord points out that BiDi has support to get a HAR direct from the browser, and we might be able to use this if it is implemented by Firefox. Accordingly, I have made the title of this ticket more generic.

[jtnord]

I am actively working towards a Java based solution using the BiDi API.

[jtnord]

How important is the actual content in the har (Vs just request/response timeline tracking)?

Currently BiDi has no way to obtain the content pending spec implementation (and then adoption in selenium)
w3c/webdriver-bidi#747 / w3c/webdriver-bidi#877

We can get the info with Chrome developer tools (DevTools) on chrome, but support for CDT was removed from the Firefox driver 2 versions ago (and it only had a partial implementation so may not have even supported this).

[timja]

I've never used the HAR recording, I just use the videos.

Doesn't matter to me.

[basil]

This seems useful to have eventually, so I think we should design for it, but it does not seem important enough to require for the first cut of the implementation.

[juliandescottes]

👋 jumping here from w3c/webdriver-bidi#877 , just wanted to mention we have https://github.com/firefox-devtools/bidi-har-export . While not Java based, it's a har recorder based on BiDi (of course missing response content, since this is not available in BiDi yet). Might be useful to you.

[jtnord]

👋 jumping here from w3c/webdriver-bidi#877 , just wanted to mention we have https://github.com/firefox-devtools/bidi-har-export . While not Java based, it's a har recorder based on BiDi (of course missing response content, since this is not available in BiDi yet). Might be useful to you.

I had found that and was already using it as inspiration :) I had written #2060 last month but ran out of time to push it forward, I have at least remembered to push my code so that someone else may be able to use it as the basis unless I get time to finish it off.