When upgrading to Fleet 4.7.0 or later, vulnerability processing is automatically enabled if
vulnerability processing and software inventory are not explicitly disabled.
If you explicitly disabled vulnerability processing, and now would like to enable this feature, first enable the software inventory feature by setting the following app config:
For example, when running a development instance of Fleet on an Apple Macbook Pro with 16 cores, matching 200,000 CPEs against the CVE database will take around 10 seconds and consume about 3GBs of RAM.
The CPU and memory usages are in burst once every hour (or the configured periodicity) on the instance that does the processing. RAM spikes are expected to not exceed the 2GBs.
There are several steps that go into the vulnerability detection process. In this section we'll dive into what they are and how it works.
The process has different parts that are more error-prone than others. Each OS and each application developer and maintainer can (and do) have their own way of defining each part of their app. Some Linux distributions are very strict, but each distribution handles things differently.
The whole pipeline exists to compensate for these differences, and it can be divided in two sections:
1. Collection:
```mermaid
graph TD;
host1[Host1 send software list]-->normalize[Normalization of names, versions, etc]
host2[Host2 send software list]-->normalize
host3[Host3 send software list]-->normalize
normalize-->store[Storage for later processing]
```
2. Processing
Processing happens in a loop. The default interval is 1hr.
```mermaid
graph TD;
interval{Once an hour}-->normalize[Normalized software list]
normalize-->downloadCPE(Download CPE database from Fleet)
This is the first step into normalizing data across platforms, as we try to get all the same data for all different types of software we detect vulnerabilities on.
Ingestion can be resource hungry, both on the hosts and the Fleet server. A lot of work has gone into reducing the resources needed, and it's still ongoing.
### Translating to CPE
With a somewhat normalized list of software, in order to search CVEs for it, we need to derive a [CPE](https://en.wikipedia.org/wiki/Common_Platform_Enumeration) from the vendor, name, version, and OS.
As described briefly above, we do this by translating the NVD database of CPEs into a [sqlite database that helps Fleet do the lookup of CPEs very quickly](https://github.com/fleetdm/nvd).
#### How accurate is this translation process?
This is the most error prone part of the process. The CPE can have some vagueness. This means that parts of it can be a `*`, which means when you match that CPE to a CVE it can match any of that part of the CPE.
If the CPE is too vague, the extreme case being all parts are `*`, all CVEs will match. You want a very specific CPE, but not too specific that a small error would make it not match a CVE (false negative).
Let's look into some examples of this stage.
#### tmux
tmux is a Unix terminal utility to multiplex ttys. It appears listed like this in macOS:
```
osquery> select * from homebrew_packages where name='tmux';
+------+----------------------------+---------+
| name | path | version |
+------+----------------------------+---------+
| tmux | /opt/homebrew/Cellar/tmux/ | 3.2a |
+------+----------------------------+---------+
```
If we look at the [official releases](https://github.com/tmux/tmux/releases/tag/3.2a) the version we get is the same as the one listed. This means that it'll be easy to map it to a CPE that will accurately represent the software.
Now let's look at Chrome on macOS:
```
osquery> select name, bundle_version from apps where name like '%Chrome%';
+-------------------+----------------+
| name | bundle_version |
+-------------------+----------------+
| Google Chrome.app | 4758.102 |
+-------------------+----------------+
```
Now things start to get slightly more tricky. We have to remove the `.app` suffix from the name, then derive the first word as the vendor and the second as the app name. We could use `bundle_name` for the app name, but nothing stops the app developer of adding the vendor to `bundle_name`, so a similar parsing would have to happen.
These are two illustrative examples. The reality is that there is no map or list of all the software available and how it's presented in each platform, so the "software to CPE" translation process is going to be evolving constantly.
### Matching a CPE to a CVE
Once we have a good CPE, we can match it against the CVE database. We download the data streams locally and match each CPE to the whole list. The matching is done using the [nvdtools implementation](https://github.com/facebookincubator/nvdtools).
### Discarding false positives
In some cases, the developers for a piece of software are also the packagers and distributors for that software. In most cases, the developers offer a set of options to install the software, but particularly in the case of Linux distributions, the package maintainers tend to be a set of different people altogether.
This is great for developers, because they don't have to be versed in all the intricacies of each package manager, but for a platform such as Fleet, trying to understand what's installed in a host, this adds an extra layer of complexity.
Linux distributions do security patches. These patches sometimes are old versions of software that is in the list of a long term maintenance distribution versions (sometimes abbreviated as LTS) with a vulnerability fix from a newer version.
In this case, the software developer might release version, say, 2.5, but in Ubuntu version 2.3-ubuntu2 is what's available. This version is not the vulnerable 2.3, but 2.3 with the patch that fixes that security vulnerability. That same fix might be version `2.4-1` in CentOS because that's what happened to be in the LTS version.
So Fleet with a pipeline up until the section above would see `some_app-2.4` and mark it as vulnerable. In reality, this is a false positive that needs to be discarded based on the data we parse for the changelogs for each distribution.
The list of potential false positives is collected in the same process as the CPE list: for each Linux distribution we support, we go through each package for that version, parse the changelogs for occurrences of CVEs mentioned. If there are any, we mark it as a potential version that Fleet might've flagged in previous steps.
If there's a match, we remove the false positive from the list.
