fleet/ee/vulnerability-dashboard/CHANGELOG.md

* Vulnerability dashboard updates by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/8

    - Added a displayName attribute to the host model.
    - Updated get-reports script to save a host's display name
    - Updated the get-remediation-timeline action to build a remediation timeline.
    - Added a list of affected hosts to the remediation timeline modal
    - Added a deploy website workflow
    - added sails-mysql dependency, updated package.json
    - updated production config

* Update CVE csv, create script to get latest CVE publish dates by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/9
    - Added a script to update the CSV of NVD publish dates

* Update dashboard graphs by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/10
    
    - Updated how the average number of days to resolve critical vulnerabilities is tracked, removed "WIP" from dashboard graphs

* Bring in latest changes to Vulnerability dashboard  by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/11
    
    - Updated this repo to have the changes from eashaw/vulnerability-dashboard:
        - Added CSV export
        - Added `teamApid` and `teamDisplayName` attributes to the `Host` model
        - Added `versionName` and `softwareName` attributes to the `VulnerabilityInstall` model
        - Updated the vulnerability list to load the first page of results after the initial page load
        - Added affected teams and affected software to the vulnerability list page.
        - Added a filter for teams to the vulnerability list page
        - Updated the CSV export to have rows for each vulnerable install
        - Updated the vulnerability-list page to support inputs via query string. e.g., `/vulnerability-list?page=2&minSeverity=4?maxSeverity=6` will load the vulnerability list page on the 2nd page of results for vulnerabilities with a CVSS score between 4 and 6.
            - Supported inputs are: `minSeverity`, `maxSeverity`, `sortBy`, `sortDirection`, `page`, and `teamApid`.

* Add changes from other vulnerability dashboard repo by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/16
    
    - disabled the is-logged-in policy by default
    - Updated the app to use the vulnerability list as default homepage

* Update get-vulnerabilities helper to include teams for resolved vulnerabilities, fix resolvedAt sort by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/19
    
    - Updated the vulnerability list page to show affected teams for resolved vulnerabilities when filtering by a team.
    - Fixed a bug where resolvedAt sorting was not working as expected.

* Add host serial number and UUID to CSV export by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/17
    
    - Added two new attributes to the `Host` model: `uuid` and `hardwareSerialNumber`.
        - Migration strategy used for these columns:
            1. Take the app offline to make sure it is not connected to the datastore. 
            2. Use a SQL query to update the `host` table to add the columns:
            ```sql
            ALTER TABLE host
            ADD COLUMN hardwareSerialNumber TEXT,
            ADD COLUMN uuid TEXT;
            ```
            3. The columns will be added to the DB table and will be set to `null` on all existing records, but since the new attributes were not defined with `allowNull: true`, the app's ORM will throw an error if it sees these values. Because we're not allowing `null` values and the new attributes are required, we'll send another query to add a placeholder value to the columns:
            ```sql
            UPDATE host
            SET `hardwareSerialNumber` = ? , `uuid` = ? 
            WHERE `uuid` IS NULL AND `hardwareSerialNumber` IS NULL;
            ```
            4. Modify the `update-reports` script locally to update Host records. e.g., In this section of the script, we modified the code that checks if a `Host` record exists for hosts sent in the API response from the Fleet instance.

            > Note: We will be adding a script to the repo for this change, and all future database model changes will be handled by scripts in the `/scripts` folder in this repo.
            ```js
            // Original code
            let newRecordsForUnrecognizedHosts = []; {
              let recognizedHosts = await Host.find({ fleetApid: { in: Object.keys(byHostFleetApidsSeenInLatestFleetScan).map((key)=>Number(key)) } });
              let unrecognizedHostApids = _.difference(Object.keys(byHostFleetApidsSeenInLatestFleetScan).map((key)=>Number(key)), _.pluck(recognizedHosts, 'fleetApid'));
              // Added code
              for (let $host of recognizedHosts){
                if($host.uuid === '?' || $host.hardwareSerialNumber === '?') {
                  await Host.updateOne({id: $host.id}).set(byHostFleetApidsSeenInLatestFleetScan[$host.fleetApid]);
                }
              }
              // Original code
              assert(unrecognizedHostApids.every((apid) => _.isNumber(apid)));
              for (let apid of unrecognizedHostApids) {
                newRecordsForUnrecognizedHosts.push(byHostFleetApidsSeenInLatestFleetScan[apid]);
              }//∞
            }//∫
            if (dry) {
              sails.log.warn(`Dry run: ${newRecordsForUnrecognizedHosts.length} hosts were seemingly enrolled. (Fleet returned them in the API.)`);
              // console.log(`would have created ${newRecordsForUnrecognizedHosts.length}:`,require('util').inspect(newRecordsForUnrecognizedHosts,{depth:null}));
            } else {
              sails.log(`Creating ${newRecordsForUnrecognizedHosts.length} host records… `);
              await Host.createEach(newRecordsForUnrecognizedHosts);
            }
            ```
            5. Modify /config/models.js to change the migration strategy to `migrate: 'safe'` and run the update-reports script while connected to the production datastore. 
    - Updated the update-reports script to:
        - Store host uuid and serial numbers
        - Remove commas found in software names and software version names
        - Trim leading and trailing whitespace from host display names

* Update CSV export and get vulnerabilities helper by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/21
    
    - Added a new input to the get-vulnerabilities helper: `includeResolvedInstalls`, a boolean that tells the helper whether or not to include resolved VulnerabilityInstall records in the affectedInstalls value for each vulnerability in the generated report. This input is currently only used when the download-vulnerabilities-csv.js calls this helper.
    - Updated variable names and comments in the get-vulnerabilities helper to match the new behavior
    - Removed the includeHostsAndSoftware input of the download-vulnerabilities-csv action and added a new input: exportType
    - The new exportType input accepts one of two strings: resolvedAndVulnerableInstalls or overview.
    - Updated download-vulnerabilities-csv to use the new input to determine what type of CSV export to build.
    - Changed the CSV export modal on the vulnerability list page:
        - Removed the "Include affected hosts and software" checkbox and added a "Report type" dropdown with two options: 'Current and resolved vulnerable software installs'(default option) and 'Overview'
        - removed the resolvedAt sort option when the "Current and resolved vulnerable software installs" report type is selected
        - Added a section to the form that gives a short description of the selected report type, and lists the columns included based on the selected Report type.
        - Added a test changes GitHub workflow.

* Update remediation timeline graphs by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/18
    
    - Updated how remediation timelines are generated.
    - Added the affected software for each host in the remediation timeline modal.

* Fix vulnerability-list page when there no vulnerabilities match filters. by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/20

    - Updated the get-vulnerabilities helper to return a `noMatchingVulnerabilities` response when it finds no vulnerabilities that match the current filter
    - Improved the empty state on the vulnerability list page

* Add vulnerability detected date to vulnerability list table by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/22
    - Added a sortable column to the vulnerability list page: "Detected date" - The date the vulnerability was first detected on the Fleet instance.
    - Updated the get-vulnerabilities action and helper to support the new sortBy value.

* Update CSV exports to use `fast-csv` by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/23
    - Added fast-csv` as a dependency

    - Updated the CSV export actions to use `fast-csv` to build CSV reports.

* Add support for filtering by a team when exporting results for a single vulnerability by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/25

    - Updated the export action in the remediation modal to create a filtered CSV report if the vulnerability list is being filtered to a team

* Update update-reports script to keep track of software updates by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/24

    - Updated the `update-reports` script to track vulnerable software upgrades, i.e., Cases where vulnerable software is updated on a host, but is affected by the same vulnerability as the previous version.

* Update update reports script to destroy Vulnerability records with no associated records by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/26

    - Updated the `update-reports` script to send a native SQL query to the app's datastore to find `Vulnerability` records that have no associated `VulnerabilityInstall` records and deletes any records found. This change is to clean up vulnerabilities that only affected a host(/hosts) that is no longer enrolled in the Fleet instance.

* Update dashboard page to use native SQL queries by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/27
    - Updated the dashboard page to use native SQL queries to get information needed to build datasets for graphs, this improves the performance (page load speed and memory usage) of the dashboard page considerably when the app is connected to Fleet instances with a large number of hosts. (When testing with a deployment of 12k hosts, 5k vulnerabilities load speeds went down 57s » 1.4s)

* Update logged-out header navigation & dashboard graph click events by @eashaw in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/28

    - Updated the header navigation to show links to the dashboard and the vulnerability list if the user is not logged in
    - Updated the severity range linked to by the clickable section of the "Number of vulnerabilities by severity" graph on the dashboard page.

* fixing sql typo that is erroring out by @stefanamaerz in https://github.com/fleetdm/fleet-vulnerability-dashboard/pull/29
    - Updated the native SQL query that finds vulnerabilities with no associated host/install records to use lowercased table names. (Thank you @stefanamaerz!)


* Keep track of specific software and vulnerabilities
  - Added a new attribute to the Vulnerability model: `isPriority`
    - Migration instructions:
    ## For Postgres
      1. Send a query to add the column to the table:
        ```
        ALTER TABLE vulnerability
        ADD COLUMN "isPriority" BOOLEAN;
        ```
      2. Send another query to update the existing database rows to add the default value
        ```
        UPDATE vulnerability
        SET "isPriority" = false
        WHERE "isPriority" IS NULL;
        ```
    # For MySQL:
      ```
      ALTER TABLE vulnerability
          ADD COLUMN isPriority BOOLEAN;

      UPDATE vulnerability
          SET isPriority = false
          WHERE isPriority IS NULL;
      ```
  - Added a new model: OperatingSystem
    - Migration instructions:
      ## For postgres
      1. Send a query to create the operatingsystem table
      ```
      CREATE TABLE operatingsystem (
          id integer NOT NULL,
          "createdAt" bigint,
          "updatedAt" bigint,
          "name" text,
          "fullName" text UNIQUE,
          "platform" text,
          "versionName" text,
          "lastReportedHostCount" INT,
          "isCompliant" BOOLEAN,
          CONSTRAINT operatingsystem_pkey PRIMARY KEY (id),
          CONSTRAINT "operatingsystem_fullName_key" UNIQUE ("fullName")
      )

      TABLESPACE pg_default;

      ALTER TABLE IF EXISTS operatingsystem
          OWNER to [USERNAME];

      CREATE SEQUENCE operatingsystem_id_seq
          INCREMENT 1
          START 1
          MINVALUE 1
          MAXVALUE 2147483647
          CACHE 1
          OWNED BY operatingsystem.id;

      ALTER SEQUENCE operatingsystem_id_seq
          OWNER to [USERNAME];

      ALTER TABLE operatingsystem
          ALTER COLUMN id SET DEFAULT nextval('operatingsystem_id_seq'::regclass);
      ```
      2. Send a query to add the operatingSystem column to the host table:
      ```
      ALTER TABLE host
          ADD COLUMN "operatingSystem" INT;
      ```
    ## For MySQL:
    1. send a query to create the operating system table:
    ```
    CREATE TABLE operatingsystem (
        id INT NOT NULL AUTO_INCREMENT,
        createdAt BIGINT,
        updatedAt BIGINT,
        name VARCHAR(255),
        fullName VARCHAR(255),
        platform VARCHAR(255),
        versionName VARCHAR(255),
        lastReportedHostCount DOUBLE,
        isCompliant BOOLEAN,
        PRIMARY KEY (id),
        UNIQUE (fullName(255)),
        UNIQUE (id)
    );
    ```
    2. Send a query to add the operatingSystem column to the host table:
    ```
    ALTER TABLE host
        ADD COLUMN operatingSystem INT;
    ```
  - Added a new model: CriticalInstall
    - Migration instructions:
      ### For Postgres
      1. Send a query to create the new table
      ```
      CREATE TABLE IF NOT EXISTS criticalinstall
      (
          "createdAt" bigint,
          "updatedAt" bigint,
          id integer NOT NULL,
          "installedAt" real,
          "versionName" text COLLATE pg_catalog."default",
          "softwareName" text COLLATE pg_catalog."default",
          platform text COLLATE pg_catalog."default",
          "fleetApid" real,
          "softwareType" text COLLATE pg_catalog."default",
          "isCompliant" boolean,
          host integer,
          CONSTRAINT criticalinstall_pkey PRIMARY KEY (id)
      )

      TABLESPACE pg_default;

      ALTER TABLE IF EXISTS criticalinstall
          OWNER to [USERNAME];

      CREATE SEQUENCE criticalinstall_id_seq
          INCREMENT 1
          START 1
          MINVALUE 1
          MAXVALUE 2147483647
          CACHE 1
          OWNED BY criticalinstall.id;

      ALTER SEQUENCE criticalinstall_id_seq
          OWNER to [USERNAME];

      ALTER TABLE criticalinstall
          ALTER COLUMN id SET DEFAULT nextval('criticalinstall_id_seq'::regclass);
      ```
    ### For MySQL
    1. Send a query to create the new table
    ```
    CREATE TABLE criticalinstall (
        createdAt BIGINT,
        updatedAt BIGINT,
        id INT NOT NULL AUTO_INCREMENT,
        installedAt BIGINT,
        versionName VARCHAR(255),
        softwareName VARCHAR(255),
        platform VARCHAR(255),
        fleetApid DOUBLE,
        softwareType VARCHAR(255),
        isCompliant BOOLEAN,
        host INT,
        PRIMARY KEY (id),
        UNIQUE (id)
    );
    ```

- Add CVE description and "Resolved in version" to the vulnerability dashboard.
  - Added a new attribute to the Vulnerability model: `cveDescription`.
    - Migration queries:
    ## For Postgres:
      ```
      ALTER TABLE vulnerability
      ADD COLUMN "cveDescription" TEXT;

      UPDATE vulnerability
      SET "cveDescription" = ''
      WHERE "cveDescription" IS NULL;
      ```
    ## For MySQL:
      ```
      ALTER TABLE vulnerability
          ADD COLUMN cveDescription longtext;

      UPDATE vulnerability
          SET cveDescription = ''
          WHERE cveDescription IS NULL;
      ```
  - Added a new attribute to the VulnerabilityInstall model: `resolvedInVersion`.
    - Migration queries:
    ## For Postgres:
      ```
      ALTER TABLE vulnerabilityisntall
      ADD COLUMN "resolvedInVersion" TEXT;

      UPDATE vulnerabilityisntall
      SET "resolvedInVersion" = ''
      WHERE "resolvedInVersion" IS NULL;
      ```
    ## For MySQL:
      ```
      ALTER TABLE vulnerabilityinstall
          ADD COLUMN resolvedInVersion VARCHAR(255);

      UPDATE vulnerabilityinstall
          SET resolvedInVersion = ''
          WHERE resolvedInVersion IS NULL;
      ```
  - Updated the `update-reports` script to add cveDescriptions to Vulnerability records.

    > Note: Existing vulnerabilities returned in the Fleet API will have this value updated automatically, but resolved Vulnerabilities will not have this information availible.

  - Updated the `update-reports` script to add `resolvedInVersion` values to VulnerabilityInstall records.

    > Note: Existing vulnerable software items returned in the Fleet API will have this value updated automatically, but previously installed vulnerable software will not be updated to have these values.

  - Updated the "/vulnerability-list" table to show resolved in versions.
  - Updated the "/vulnerability-list" to open a modal containing the description of a CVE when the CVE ID on the table is clicked.
  - Updated the CVS export actions to include CVE descriptions and resolved in versions.

* Add support for Okta SSO
  - Added two new dependencies: `@okta/oidc-middleware@4.0.1` and `@okta/okta-sdk-nodejs@^3.2.0`
  - Removed the `sails-hook-sockets` dependency
  - Added optional support for Okta SSO. To use this, three config variables are required:
    1. `sails.config.custom.oktaOrgUrl`: The URL of your organizations Okta account
    2. `sails.config.custom.oktaClientId`: The client secret of the application that has been created for this app in Okta.
    3. `sails.config.custom.oktaClientSecret`: The client secret for the application that has been creeated in Okta.
    Once these values are set, the app will automatically disable the built in authentication system and replace it with Okta SSO.

    > When using the app with Okta SSO, the following pages are automatically disabled in the app: `/account`, `/login` (Note: the built-in login page for route is overridden by the Okta middleware).
  - *Important* Changed the default policies configuration to make the `is-logged-in` policy apply to every route except `/logout`. To disable the `is-logged-in` policy globally, you can override the policies config in your `config/env/production.js` or through a config variable e.g., `sails_policies={"*":true}`

* Update `update-reports` script #42
  - Updated the update-reports script not to throw an error if a host is detected with an operating system not returned in the API response from the /os_versions endpoint. Now, if a host has been updated after the operating systems in the API response, it will log a message about the host and continue the script without reporting the new status of that host. (They will be updated on the next run after the OS host counts are updated.) If the host was updated before the OS hosts count, the script will throw an error.

* Track patch progress for CVEs that have not been reported by the Fleet instance #46
  - Added an attribute to the `Platform` model: `priorityCveIds`. A JSON array of CVE IDs.
    - Migration queries:
      ### For postgres:
        ```
        ALTER TABLE platform
          ADD COLUMN "priorityCveIds" json;
        ```
      ### For MySQL:
        ```
        ALTER TABLE platform
          ADD COLUMN `priorityCveIds` JSON;
        ```
  - Updated `view-welcome` to report Patch progress for CVE IDs that are only present in the `Platform` record, and to update any `Vulnerability` records that previously only existed in the `Platform` record.
  - Updated the `update-priority-vulnerabilities` action to:
     - Throw an error if a CVE ID is not in the expected format
     - Track patch progress for CVE IDs that don't match a `Vulnerability` record, but are a valid CVE Id.
     - Update the JSON array of CVE IDs in the `Platform` record.
  - Updated `bootstrap.js` to throw an error if more than one `Platform` record exists in the database, and to create a Platform record if none are found.
  > **Important**: If the `platform` table in the connected database does not have the `priorityCveIds` column, the app will display an error and will not start. Please use the migration queries above before starting the app with this change.
  - Updated empty states on the priority CVE table. ("**N/A**" » "**-**")