npm install webdriver is not working properly: 3 solutions

So, you thought about using locally installed Microsoft Edge (chromium) or Google Chrome as automaton for your nodeJS project? Bad luck, the webdriver versions available on npm for Windows do not match the browser’s MAJOR version… That’s a FAIL! Here is how to circumvent this problem.

The process of  coming up with an automated solution is all that matters to me, and I felt like sharing it since it gave me so much trouble and frustration.

What is a webdriver?

WebDriver allows you to automate Microsoft Edge and Chrome by simulating user interaction. Tests that use WebDriver have some advantages over JavaScript unit tests that run in the browser:

• WebDriver accesses functionality and information that’s not available to JavaScript running in browsers.
• WebDriver simulates user events or OS-level events more accurately than JavaScript unit tests.
• WebDriver manages multiple windows, tabs, and webpages in a single test session.
• WebDriver runs multiple sessions of chromium on a specific machine.

To automate chromium with WebDriver to simulate user interaction, you need three components:

1. Microsoft Edge or Google Chrome:
   installed by the user, and version installed is controlled by the provider. You have no say in which version is installed unless you also find a way to download archived installers. Or use portable versions.
2. Microsoft Edge WebDriver:
   pulled by npm as a dev dependency or manually pulled as a global module. In either case, you are not in control of the version installed anymore since version 91.x –
3. A WebDriver testing framework:
   Selenium comes to mind. And also: Cypress, Puppeteer, TestCafe, Playwright and Geb.

I will only mention Chrome and Microsoft Edge moving forward, but the same is true for Mozilla Firefox.

Critical point: MAJOR version of the driver MUST match the MAJOR version of the browser.

Description of the Problem, as of today 5/4 2023

I mention the date, because the problem did not happen until that date, when I had to update the modules.

package.json

For the sake of simplicity, here is the package.json of my project, nothing fancy here:

{
  "name": "projectX",
  "version": "2.8.89",
  "license": "MIT",
  "description": "modules required for projectX",
  "scripts": {
    "test": "mocha -g testtt ..\\test"
  },
  "private": true,
  "devDependencies": {
    "cleanup": "^0.3.0"
  },
  "dependencies": {
    "axios": "^1.4.0",
    "chai": "^4.3.7",
    "chai-as-promised": "^7.1.1",
    "chromedriver": "^112.0.1",
    "colors": "^1.4.0",
    "fast-csv": "^4.3.6",
    "geckodriver": "^3.2.0",
    "jasmine": "^4.6.0",
    "mocha": "^10.2.0",
    "mocha-chrome": "^2.2.0",
    "msedgedriver": "^91.0.0",
    "puppeteer": "^20.1.0",
    "requirejs": "^2.3.6",
    "selenium-side-runner": "^4.0.0-alpha.46",
    "selenium-webdriver": "^4.9.0",
    "util": "^0.12.5"
  }
}

The way it works: you add global modules with npm install -g <module ..> and then you add dev dependent modules with npm install <module ..>; those are the ones that will be referenced with their version in package.json.

Notice how many of the versions above have no PATCH value, most end with a 0. npm is smart enough to get the latest PATCH version that matches the MAJOR+MINOR that you need, when you install those modules.

This is where it gets wild: notice how as of 5/4/2023, npm major version for msedgedriver is 91.0.0? Do you think msedgedriver version 91 will work with current Microsoft Edge, which is 113.0.1774.35 ?? Answer is: no.

Same for Chrome: all npm can install is 112.x while Chrome/Chromium/Opera/etc are also available only in MAJOR version 113 to that day. They are incompatible!

Here are some solutions to consider:

How to Match webdriver MAJOR version to browser version in 2023

Again, I will only discuss chrome and msedge, but the same is probably true for Firefox.

Solution 1: do not upgrade (impossible)

Sounds stupidly easy but worth mentioning. I personally stay with Electron 12 for those reasons, to get less frustration during development. Once features become officially deprecated or a security risk, then it’s time to update the code.

Unfortunately, this is now impossible. The browsers you install do upgrade themselves automatically. Both Edge and Chrome even have the audacity to create two Windows scheduled tasks to keep up to date and send “information” to the mother ship. Restart Chrome tomorrow and good chances are that the version has changed again.

For this to work, simply use a portable version of msedge or chrome.

Solution 2: post-install the needed version of the driver with npm

Just so you know, here are the related commands to know by heart with nodeJS:

• to install modules, latest version: npm install <module>
• to update modules to the latest, in package.json: ncu -u
• after the upgrade with ncu, you still have to install those modules with npm install (no arguments)

So, let’s say you do all this as of May 2023, and… now the webdriver is incompatible with your browser! Stackoverflow have only two similar answer to this problem: npm install the webdriver with specific versions as parameter.

npmjs has the answer for msedge:

npm install msedgedriver --edgechromiumdriver_cdnurl=https://msedgedriver.azureedge.net/ --edgechromiumdriver-force-download --edgechromiumdriver_version=112.0.1722.68

For chrome, npmjs has a similar page with similar commands:

npm install chromedriver --detect_chromedriver_version --chromedriver-force-download

So, the process is this when you update your modules: after npm install, all you get is msedgedriver 91.x, then you execute the command above with the version needed, and the driver version 91.x gets replaced by 112.0.1722.68 … right??

Except… this may fail. When you specify an exact version for download, this version has to exist on the servers of our friends at Microsoft/Google. And on 5/4/2023 when I had to do that, msedgedriver version 112.0.1722.68 was NOT present for download. The latest version available was 112.0.1722.64

The next day or 3 days later, can’t remember, then 112.0.1722.68 was finally available. Yeah, your success depends on you LUCK in NOT upgrading your modules when there is a discrepancy of versions available between browser and driver 🙂

So, if you followed the thought process, you can see how there is a problem here: you cannot AUTOMATE this update anymore.

Today 5/10/2023 the npm commands would likely work if you get the LATEST version from the url above and plug it in the command parameter, but 6 days ago, and likely in the future the problem will occur again.

Solution 3: foolproof way to update the correct webdriver version

There is another way, yes. It’s not complex enough, there HAD to be another way, yay!

The way is this:

1. Get the LATEST available webdriver version
2. Plug it in a curl download command
3. unzip it to overwrite the faulty driver

Seems old school, seems wrong… But that’s the ONLY way to get a working, matching webdriver for a browser which you do not control the version anymore. Seems stupid to me, to say the least, but it’s automated in a batch script. I can sleep well, knowing that when I upgrade my modules tomorrow with a browser that also has been upgraded without my consent, well, everything will work together!

With msedge, you get a file encoded in UTF16-LE BOM, so you will need to convert the output to ASCII or UTF8 with another tool… So much work…

Below are the batch script functions I use to update my webdrivers. There are more functions I use to gather the browser version but for your sake, I tried to gather everything together.

Batch script to update msedge webdriver to match the browser installed:

Requisites:

• 7-zip (at least 7z.exe and 7z.dll)
• busybox
• curl (powershell or wget would work too)

:getMSEDGEDriverVersion
REM for msedge the command is:
REM D:\projectX\x64\node_modules\msedgedriver\lib\msedgedriver\msedgedriver.exe -version
REM and you get this output:
REM Microsoft Edge WebDriver 109.0.1518.49 (a0ba3a09d1b75211882758182c96ccb56aec7b7d)

for /f "usebackq tokens=1-4" %%a in (`call D:\projectX\x64\node_modules\msedgedriver\lib\msedgedriver\msedgedriver.exe -version 2^>NUL`) DO (
set driverVersion=%%d
)

for /f "tokens=1 delims=." %%v in ("%driverVersion%") DO set driverVersionMajor=%%v
goto :EOF


:browserDriverUpdate_edge browserVersionToGet
pushd %NODE_PATH%\..

REM :: https://stackoverflow.com/questions/71620168/is-there-a-link-to-get-the-latest-microsoft-edge-version-number
REM :: get latest stable version because driver version may not match binary
REM :: therefore we cannot rely on browserVersionToGet anymore
REM :: also we get ��112.0.1722.68 which is UTF-16LE BOM encoded
for /f %%a in ('curl -sSL https://msedgedriver.azureedge.net/LATEST_STABLE --output - ^| busybox iconv -c -f UTF-16LE -t ASCII') DO set edgechromiumdriver_version_latest=%%a

:: Now, this is not sufficient to get the latest driver version because it may not be available for download.
:: Example: on 2023-05-04 we get binary=112.0.1722.68 and latest driverVersion=112.0.1722.68 but ...
:: what's available at https://msedgedriver.azureedge.net/ is at most 112.0.1722.64
:: What do we do now??

REM :: https://www.npmjs.com/package/msedgedriver?activeTab=readme
REM :: same shite as with chromedriver: cannot get npm to download the correct version anymore
REM IF DEFINED VERBOSE call echo VERBOSE %b%call npm install msedgedriver --edgechromiumdriver_cdnurl=https://msedgedriver.azureedge.net/ --edgechromiumdriver-force-download --edgechromiumdriver_version=%edgechromiumdriver_version_latest% %END%
REM call npm install msedgedriver --edgechromiumdriver_cdnurl=https://msedgedriver.azureedge.net/ --edgechromiumdriver-force-download --edgechromiumdriver_version=%edgechromiumdriver_version_latest% >NUL

call curl -sSL https://msedgewebdriverstorage.blob.core.windows.net/edgewebdriver/%edgechromiumdriver_version_latest%/edgedriver_win32.zip --output %TEMP%\edgedriver_win32.zip

REM :: https://stackoverflow.com/questions/69439276/npm-install-chromedriver-is-not-working-properly
REM :: solution to this nonsense:
7z.exe e -y "%TEMP%\edgedriver_win32.zip" -so msedgedriver.exe -r >"D:\projectX\x64\node_modules\msedgedriver\lib\msedgedriver\msedgedriver.exe"

popd

goto :EOF

Not shown above, but you also want to compare the MAJOR version of the driver to the browser, so I included the function that gets you the driver’s value. Everything here is also valid for Linux and MacOS environments. Just translate the batch to nash, nothing difficult here.

Batch script to update chrome webdriver to match the browser installed:

:getCHROMEDriverVersion browser
REM for chrome the command is:
REM D:\projectX\x64\node_modules\chromedriver\lib\chromedriver\chromedriver.exe -version
REM and you get this output:
REM ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414@{#1172})

for /f "usebackq tokens=1-4" %%a in (`call D:\projectX\x64\node_modules\chromedriver\lib\chromedriver\chromedriver.exe -version 2^>NUL`) DO (
REM :: for Chrome, version output format is different from edge, go figure
set driverVersion=%%b
)

for /f "tokens=1 delims=." %%v in ("%driverVersion%") DO set driverVersionMajor=%%v

goto :EOF


:browserDriverUpdate_chrome browserVersionToGet
pushd %NODE_PATH%\..

:: https://swimburger.net/blog/dotnet/download-the-right-chromedriver-version-and-keep-it-up-to-date-on-windows-linux-macos-using-csharp-dotnet
:: get the latest chrome stable driver and download it from https://chromedriver.storage.googleapis.com/index.html?path=113.0.5672.63/
for /f %%a in ('curl -sSL https://chromedriver.storage.googleapis.com/LATEST_RELEASE --output -') DO set chromedriver_version_latest=%%a

:: since 2023-05-04 same problem with Chrome: binary is 1 major ahead of the driver: 113.0.5672.63 # 112.0.5615.49
:: therefore we cannot rely on detect_chromedriver_version at all
REM REM :: https://www.npmjs.com/package/chromedriver
REM IF NOT DEFINED chromedriver_version_latest call :warning %~0 Could not get LATEST_STABLE from https://chromedriver.storage.googleapis.com/LATEST_RELEASE

REM IF DEFINED VERBOSE call echo VERBOSE %c% call npm install chromedriver --detect_chromedriver_version --chromedriver-force-download %END%
REM call npm install chromedriver --detect_chromedriver_version --chromedriver-force-download >NUL

call curl -sSL https://chromedriver.storage.googleapis.com/%chromedriver_version_latest%/chromedriver_win32.zip --output %TEMP%\chromedriver_win32.zip

REM :: this shite does not work: zipfile is ignored
REM IF DEFINED VERBOSE call echo VERBOSE %b%call npm install chromedriver --chromedriver_filepath=%TEMP%\chromedriver_win32.zip %END%
REM call npm install chromedriver --chromedriver_filepath=%TEMP%\chromedriver_win32.zip >NUL

REM :: https://stackoverflow.com/questions/69439276/npm-install-chromedriver-is-not-working-properly
REM :: solution to this nonsense:
7z.exe e -y "%TEMP%\chromedriver_win32.zip" -so chromedriver.exe -r >"D:\projectX\x64\node_modules\chromedriver\lib\chromedriver\chromedriver.exe"

popd

goto :EOF

Similar functions for chrome, and probably similar for Firefox which I do not use for testing.

Conclusion: Back to the Old Age of doing stuff manually?

EDIT: as of today 5/10/2023, chromedriver is finally available to npm in version 113:

Now, tell me that I hallucinate, tell me something I am doing wrong… Because I want to understand. This is non-sense. Back to the old age of downloading stuff manually with quirks and complexity, instead of using npm. What can I say?

How do you do? Do you have a better solution? Please share in the comments below and have a wonderful day!