Context

When working in a restricted environment, it can be heavily restricted whereby only packages from the whitelisted domains can be installed.

This results in breakage in your pipeline as the installation of these packages has been blocked by your corporate firewall.

Signs of being blocked

1. You will observe [ECONNRESET errors](stackoverflow.com/questions/18419144/npm-no..) appear when attempting to run `npm install`.

2. You will also see NPM attempting to use `node-gyp` to compile the packages, which will also similarly fail to do so.

   • Refer to [this post](stackoverflow.com/questions/52535095/how-to..) for more explanation.

3. To check the installation source of the NPM package, you can check the installation source using the following link `registry.npmjs.com<package_name>`.

   • E.g. `registry.npmjs.com/msnodesqlv8`. Search for the keyword `url`

---

Execution Process

Tech Stack & Considerations

NPM does not allow defining custom-named package.json, hence there is a need to temporarily remove those packages from package.json file.

I will be using Linux built-in commands and jq to execute a list of steps to allow the installation of both online and offline packages.

The setup used in this illustration assumes

• All required files would be under the same working directory

• The packages have been prepared and archived into a tar file named node_modules.tar .

• The packages that cannot be installed would be cpu-features and jsdom

  • Remember this is just an illustration, may not be true in your context!

This will be the sample package.json file I will be working with:

{
  "name": "sample app",
  "version": "1.0.0",
  "description": "sample app",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "test user",
  "license": "ISC",
  "dependencies": {
    "axios": "^1.2.2",
    "cpu-features": "^0.0.4",
    "jsdom": "^19.0.0",
    "print-message": "^3.0.1"
  }
}

Walkthrough

Order matters!

Online packages must be installed first because when npm install is executed, NPM will attempt to pull all packages from the Internet.

By removing the packages that can't be directly installed first in the filtered `package.json`, you will ensure that all packages that can be pulled from the Internet will be installed first.

Subsequently, when installing offline packages, the --offline option will ensure no network requests will be done during installation.

1. Install online packages

Steps to perform

1. Create a backup of package.json

2. Get the package names to exclude from package.json

3. Remove the packages using jq

4. Do installation with npm install for online packages

# Create a backup of package.json
cp "package.json" "package.json.bak"

# Store the packages name in this format to ensure jq can parse it
offlinePkgsName='"cpu-features","jsdom"'
jqQuery="del(.dependencies[$offlinePkgsName])"

# Define the query to use with jq
jq "$deletejqQueryString" "$package.json" > "package-temp.json" && mv "package-temp.json" "package.json"

npm install

2. Install offline packages

Steps to perform

1. Rename the backup of package.json to the original name

2. Extract your self-hosted packages from the tar file

   • There shouldn't be any folder clashing with the packages folder name

   • To be on the cautious side, you can include -k option for tar to retain old files without replacing old files.

3. Perform an offline installation

   • Refer to [NPM documentation](docs.npmjs.com/cli/v9/using-npm/config#offl..) for the `--offline` option

# Rename the backup of package.json to be the original file again
mv "package-temp.json" "package.json"

# Extract your self-hosted packages
tar -xvfk "node_modules.tar"

# Perform offline installation
npm install --offline

Caveat

Please note that the following approach only works when you build and deploy your application on the same OS architecture (i.e. build in Linux environment, deploy in Linux environment).

If not, you will face the error of Invalid ELF header when you attempt to run your unit tests (e.g. Jest) in your pipeline. You may not detect it until you run your test in interactive mode.

Closing off

No doubt, there are existing NPM packages out there that help to do the following such as `partial-install` or `npm-install-offline`. However, it seems that they are not regularly maintained. 😢

Hope these steps give you insight into doing an installation for the combination of online and offline packages with built-in Linux commands and jq (JSON parser)! 🙌🏼

Cheers 🍻