Commit Graph

61 Commits

Author SHA1 Message Date
Juan Fernandez
009a87d33e
Feature 10196: Add filepath to end-points and third party integrations (#11285)
Adds the software installed path property to the proper end-points and third party integrations (webhook, Zendesk and Jira).
2023-05-17 16:53:15 -04:00
Juan Fernandez
7f83135aa1
Feature: Store installed file path when ingesting software (#11214)
Store software installed paths into the host_software_installed_paths table when ingesting osquery software data.
2023-05-17 14:49:09 -04:00
Juan Fernandez
d353a3deb4
Handle flaky vulnerability tests (#11262)
- Refactored some of the vulnerabilities tests to help with flakiness.
- Don't load NVD assets if local assets have a timestamp of today.
2023-04-21 19:37:29 -04:00
Juan Fernandez
4c2ddba2e4
Clean out-of-date NVD results. (#10514)
Keep the vulnerabilities detected via NVD and stored in the DB in sync. with the results from the NVD vulnerability process.
2023-04-03 13:45:18 -04:00
Juan Fernandez
56ed2727b5
Updated translation rules so that Docker Desktop can be mapped to the proper CPE (#10326)
Updated translation rules so that Docker Desktop can be mapped to the proper CPE.
2023-03-09 17:46:57 -04:00
Juan Fernandez
6ff2c449cf
Improve test coverage and bug fix (#10301)
Addresses https://github.com/fleetdm/fleet/issues/10112

Improved test coverage, also fixed a bug related to vulnerability processing for mac office apps.
2023-03-06 15:07:27 -04:00
Juan Fernandez
7e366272c0
Feature 9386: Parse the Mac Office release notes for vulnerability processing (#9993)
This PR adds the capability of parsing the release notes posted in https://learn.microsoft.com/en-us/officeupdates/release-notes-office-for-mac into a JSON metadata file (to be released in the NVD repo) and use it for detecting vulnerabilities on Mac Office apps.
2023-02-24 14:18:25 -04:00
Benjamin Edwards
0db432f7e3
vuln processing distinct command (#9813)
closes https://github.com/fleetdm/fleet/issues/3723

Add new vuln processing command, configs, and documentation on how to utilize.
2023-02-17 10:00:57 -05:00
Juan Fernandez
e376d53294
Bug 9386: Docs chrome extension has the wrong CPE. (#9524)
Related to #9386 - this should fix one of the three reported problems.

* Add the ability to add exclusion rules to cpe_translations.
* Added exclusion rule for Docs chrome extension.
2023-01-27 15:11:20 -04:00
Lucas Manuel Rodriguez
f64ba83b2e
Make test more stable (#9162) 2023-01-03 16:39:28 -03:00
Lucas Manuel Rodriguez
ac22aadc13
Fleet server and tooling to use NETWORK_TEST_GITHUB_TOKEN when environment variable is set. (#9143)
* WIP

* Add more logging

* Check rate limit at end of action

* Add github client in more places

* Add new published firefox 93 vulnerabilities to tests

* Remove fmt printfs

* Restore CI check settings

* Readd newline
2023-01-03 14:56:11 -03:00
gillespi314
94dd1c3745
Ingest pending MDM hosts (#9065)
Co-authored-by @roperzh
2022-12-26 15:32:39 -06:00
gillespi314
6fb3a87ae9
Enable errcheck linter for golangci-lint (#8899) 2022-12-05 16:50:49 -06:00
Roberto Dip
837fef4bc4
ensure we provide a future date in MSRC test (#8897)
This test was failing in Dec 2022, because in this line:

```go
_, err := sut.GetFeed((now.AddDate(0, 1, 0)).Month(), now.Year())
```

`(now.AddDate(0, 1, 0)).Month()` returns `"January"` , and `now.Year()` returns `2022` , so we were sending a date in the past.
2022-12-01 11:29:07 -03:00
Michal Nicpon
9ad1721efd
fix issue with duplicate vulns detected using nvd (#8613)
The OVAL analyzer falsely assumes that any vulnerabilities detected on a
host only come from OVAL. However, it is possible that NVD detects
vulnerabilities on these hosts even though it excludes software from
deb_packages and rpm_packages. For example, a python package twisted
v22.20 has a vulnerability CVE-2022-39348 detected by NVD. The OVAL
analyzer would delete this vulnerability, and it would be re-inserted by
the NVD scanner on the next run. This creates a loop.

The fix is to only delete vulnerabilities that are actually detected
using OVAL. We already store this in the source column in the
software_cve table.
2022-11-10 10:28:00 -07:00
Juan Fernandez
53e112d264
Feature 7494: Use the MSRC security bulletin artifacts for detecting Win OS vulnerabilities (#7889)
Use the MSRC security bulletin artifacts for detecting Win OS vulnerabilities
2022-10-28 11:12:21 -04:00
Juan Fernandez
9444016571
Fixed bug with CPE matching, UTM.app was matching to the incorrect CPE (#8149)
Fixed bug with CPE matching, UTM.app was matching to the incorrect CPE.
2022-10-24 12:46:18 +00:00
Juan Fernandez
582772fbee
Fixed bug with duplicated vulnerabilities (#8089)
Previous code assumed CVEs were unique per NVD file, which was   incorrect.
2022-10-05 19:48:37 -04:00
Martin Angers
ec75fb10b2
Fix CI tests for Go (#8079) 2022-10-04 15:48:14 -04:00
Juan Fernandez
812d3c85de
Fixes various bugs with NVD vulnerability detection (#7963)
- Improved NVD CPE matching process.
- Fixed bug with the 'software/<id>' endpoint not showing the generated_cpe value.
2022-10-04 07:04:48 -04:00
Tomas Touceda
8457e55b53
Bump go to 1.19.1 (#7690)
* Bump go to 1.19.1

* Bump remaining go-version to the 1.19.1

* Add extra paths for test-go

* Oops, putting the right path in the right place

* gofmt file

* gofmt ALL THE THINGS

* Moar changes

* Actually, go.mod doesn't like minor versions
2022-09-12 20:32:43 -03:00
Juan Fernandez
98e4d6e0bc
Feature 7394: Use MSRC parser to generate security bulletin artifacts (#7491)
Generate security artifacts using the MSRC parser.
2022-09-12 15:17:13 -04:00
Michal Nicpon
0709d1bc5c
improve vuln cpe matching on macos (#6985)
* add cpe translations
* fix matching on target_sw
2022-09-01 10:02:07 -06:00
Juan Fernandez
2699c22143
Feature 7077: Add MSRC feed parser (#7424)
Added parser for MSRC
2022-08-30 16:39:50 -04:00
Juan Fernandez
3048a07fd1
Feature 7076: Ingest installed windows updates (#7138)
* Ingest installed Windows updates and store them in the windows_updates table. 
* Added config option for enabling/disabling Windows update ingestion and Windows OS vuln. detection.
2022-08-26 14:55:03 -04:00
Juan Fernandez
0fab2fe0eb
Feature 6538: Remove cpe_id from the software_cve table (#6587)
* Don't insert dummy values for non-match CPEs, exclude OVAL platforms when getting software iterator.
* Removed CPEID from SoftwareVulnerability type and software_cve.cpe_id column.
* Removed GeneratedCPEID property
2022-08-24 13:10:58 -04:00
Juan Fernandez
966bfbf85e
Feature 6487: Deprecate cpe_id from software_cve table (#6562)
Part 2/3 of the removal of the cpe_id column from the software_cve table in favor of using the newly added software_id coumn.
2022-08-04 09:24:44 -04:00
Juan Fernandez
af80406b59
Bug 6777: Make the regexp used for detecting the oval platform major, min versions more permissive (#6800)
When getting the major/minor version for OVAL processing, handle the LTS post-fix for Ubuntu platforms
2022-07-21 12:55:07 -04:00
Juan Fernandez
5b90d11c85
Bug 6508: Don't show non-CVE entries (#6686)
Only include vulnerabilities (CVEs) in Fleet UI and API
2022-07-18 09:23:32 -04:00
Juan Fernandez
79bf51b03c
Feature 6242: Use oval to detect vulnerabilities on Fedora hosts (#6330)
* Feature 6242: Scan Fedora hosts using OVAL definitions
2022-06-24 11:02:51 -04:00
Juan Fernandez
9d01ba33c6
Feature 6096: Scan RHEL/CentOS hosts using OVAL definitions (#6241)
Extended the OVAL parser/analyzer so that we can scan RHEL based systems.
2022-06-23 16:44:45 -04:00
Eng Zer Jun
abe33f1d8d
test: use T.TempDir to create temporary test directory (#6080)
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-06-13 10:20:38 -03:00
Lucas Manuel Rodriguez
4cfeaa1580
Do not use golangci action for better reproducibility (use make lint-go) (#6175)
* Do not use golangci action for better reproducibility

* Add fix to trigger build

* Fix all reported issues

* fix more lint errors

* Add missing import

* Remove unused method

* Remove change not necessary
2022-06-10 18:52:24 -03:00
Lucas Manuel Rodriguez
3dd89bc0a1
Fix flaky test TestVulnerabilityDataStream (#6162) 2022-06-09 10:36:37 -03:00
Michal Nicpon
bac168e3fd
fix mem usage cve sync (#6131) 2022-06-08 13:15:44 -06:00
Juan Fernandez
337734fce1
Bug: Issue while parsing dpkg objects with inline names (#6146)
* Bug: Issue while parsing dpkg objects with inline names

Dpkg objects can also define their names inline, not only using variable references.
2022-06-08 14:29:58 -04:00
Juan Fernandez
ef73039559
Improve vulnerability detection for Ubuntu (#6102)
Feature: Improve our capability to detect vulnerable software on Ubuntu hosts

To improve the capability of detecting vulnerable software on Ubuntu, we are now using OVAL definitions to detect vulnerable software on Ubuntu hosts. If data sync is enabled (disable_data_sync=false) OVAL definitions are automatically kept up to date (they are 'refreshed' once per day) - there's also the option to manually download the OVAL definitions using the 'fleetctl vulnerability-data-stream' command. Downloaded definitions are then parsed into an intermediary format and then used to identify vulnerable software on Ubuntu hosts. Finally, any 'recent' detected vulnerabilities are sent to any third-party integrations.
2022-06-07 21:09:47 -04:00
Michal Nicpon
1044182b7b
fix epss probability (#6083) 2022-06-03 11:37:47 -06:00
Michal Nicpon
30a864c8d2
Sync CVE scores periodically (#5838) 2022-06-01 10:06:57 -06:00
Michal Nicpon
983f9e59b3
Include CVE scores when listing software (#5673) 2022-05-20 10:58:40 -06:00
Lucas Manuel Rodriguez
114b678e26
test: Add retry to the correct part that has sync enabled (#5688) 2022-05-11 10:11:51 -03:00
Lucas Manuel Rodriguez
fda79a8770
Run network test serially to prevent timeouts on Github CI (#5557)
* Run network test serially to prevent timeouts on Github CI

* Revert lint changes

* Add simple file lock

* Revert test change

* Clarify error check
2022-05-10 11:52:33 -03:00
Martin Angers
36702ede8d
Make recent vulnerabilities max age configurable. (#5081) 2022-04-12 14:48:15 -04:00
Tomas Touceda
f2aba83a73
Extend vulnerability age to 30 (#4901) 2022-04-06 11:42:02 -03:00
Michal Nicpon
779623934c
Fix update invite email conflict (#4751) 2022-03-28 09:38:57 -06:00
Martin Angers
80079e4fd7
Fix race in mock datastore for recent_vulns test (#4230) 2022-02-16 09:23:27 -05:00
Lucas Manuel Rodriguez
be72dc356c
Add CentOS parsing+post-processing to reduce false positives in vulnerability processing (#4037)
* Add CentOS parsing and post-processing in fleet

* Add tests and amend SyncCPEDatabase

* Add test for centosPostProcessing

* Changes from PR comments

* Amend software test

* Fix sync test

* Add index to source and vendor

* Use os.MkdirTemp

* Rearrange migrations

* Regenerate test schema

* Add support for testing migrations (#4112)

* Add support for testing migrations

* Rename migration in tests

* Changes suggested in PR

* Go mod tidy
2022-02-14 15:13:44 -03:00
Tomas Touceda
c45115a915
Properly handle path in CVE URL prefix (#4174) 2022-02-11 14:10:13 -03:00
Martin Angers
6e2ba62744
Trigger webhooks for recently published vulnerabilities (#3941) 2022-02-02 16:34:37 -05:00
Tomas Touceda
b90e2e2e3d
Issue 3901 match target sw (#3982)
* Allow to search with target_sw with a period

* Remove tests for a different thign

* Add a test for programs as a source as well

* Use MATCH again and add proper tests
2022-02-02 17:17:41 -03:00