Commit Graph

337 Commits

Author SHA1 Message Date
Zachary Wasserman
06ed4c2771 Only calculate online interval once when retrieving hosts (#1457)
This makes a major perf difference, as it reduces the O(# of hosts) queries
down to constant.
2017-03-30 08:30:05 -07:00
Victor Vrantchan
a9f5442cd2 use stdlib context in go-kit methods (#1454) 2017-03-29 11:46:52 -04:00
John Murphy
693600ba2b Decorator support (#1430)
* Added migrations

* Added handler for decorators

* Added logging and metrics for decorators

* WIP decorators

* Wip added decorator service

* Added service implementation

* Added mock decorator

* Added modify decorator

* Added testing

* Addressed code review issues raised by @zwass

* Added logging for missing type per @zwass
2017-03-28 16:45:18 -05:00
John Murphy
039e9e1a98 Add TLS profiles to command line (#1444)
* Add TLS profiles to command line

* Code review changes per @groob

* fixed busted test
2017-03-27 23:21:48 -05:00
John Murphy
71e66e6d16 Added code to trim whitespace and trim trailing slash from input server url (#1442) 2017-03-22 14:40:01 -05:00
Zachary Wasserman
55a1e3ec63 Bump default SELECT limit in MySQL datastore (#1440)
This is a temporary workaround for #1431.
2017-03-21 12:13:12 -07:00
John Murphy
404afe3e22 Fix issue where config interval can be number or string (#1432)
* Fix issue where config interval can be number or string

* Implemented @groob code review suggestions

* Added type assertions with graceful failure if something slips through validation

* Implemented code review changes per @zwass
2017-03-21 12:02:13 -05:00
Zachary Wasserman
b59cd2b48b Accelerate checkins when hosts enroll (#1423)
Return `accelerate: 10` with distributed queries if we do not have host
details. This facilitates the host quickly joining all expected labels, as
`platform` gated label queries will not be returned until the detail queries
return with the platform.

Fixes #1421.
2017-03-21 09:17:38 -07:00
Zachary Wasserman
da096d7b50 Fix platform detection for CentOS6 (#1415)
On CentOS6 there is a bug in which osquery incorrectly reports an empty string
for platform. This PR fixes our detection of centos in this case.

Fixes #1339
2017-03-15 15:40:18 -07:00
Zachary Wasserman
715d908613 Update go-kit to 0.4.0 (#1411)
Notable refactoring:
- Use stdlib "context" in place of "golang.org/x/net/context"
- Go-kit no longer wraps errors, so we remove the unwrap in transport_error.go
- Use MakeHandler when setting up endpoint tests (fixes test bug caught during
  this refactoring)

Closes #1411.
2017-03-15 08:55:30 -07:00
Zachary Wasserman
cc99f081fd Add a minimum threshold for the expected host checkin interval (#1397)
Adding this minimum helps prevent flapping of the online status.

Closes #1396.
2017-03-13 08:55:10 -07:00
Zachary Wasserman
5781613249 Minor fixes in email footer (#1394) 2017-03-10 15:18:40 -08:00
Kyle Knight
9791e85881 Add styles to SMTP setup confirmation email (#1382) 2017-03-10 17:02:45 -05:00
Zachary Wasserman
b4e40cf466 Warn before running migrations (#1385)
- Refactor MigrationStatus() to return relevant info
- Warn before running migrations

Closes #1368
2017-03-09 10:40:52 -08:00
Mike Arpaia
f510691ad0 Fix host status regression from #1334 (#1383) 2017-03-09 11:36:24 -07:00
Zachary Wasserman
84ffd1d5a3 Update MySQL definitions for Unicode support 🔥 (#1360)
- Set default database character set to utf8mb4
- Convert character sets for each table to utf8mb4
- Use utf8mb4 as charset in connection string

Closes #1268
2017-03-08 09:29:25 -08:00
Zachary Wasserman
343262b799 Warn on startup if migrations not completed (#1353)
- Add datastore method to check migration status.
- Print a warning on startup if migrations are not completed.

Fixes #1279
2017-03-08 09:17:07 -08:00
Mike Arpaia
e4db95d2b5 More sensible online status calculation (#1334)
Improve the mechanism used to calculate whether or not hosts are online.

Previously, hosts were categorized as "online" if they had been seen within the past 30 minutes. To make the "online" status more representative of reality, hosts are marked "online" if the Kolide server has heard from them within two times the lowest polling interval as described by the Kolide-managed osquery configuration. For example, if you've configured osqueryd to check-in with Kolide every 10 seconds, only hosts that Kolide has heard from within the last 20 seconds will be marked "online".
2017-03-07 19:47:51 -07:00
Victor Vrantchan
79f2528419 provide context for ServeFrontend 500 errors (#1349)
Closes #1343
2017-03-03 19:49:55 -05:00
Zachary Wasserman
fa84415c13 Support osquery logger_tls_compress flag (#1346) 2017-03-03 14:43:11 -08:00
Victor Vrantchan
eff7ffa44f rotate osqueryd logs on SIGHUP (#1316)
rotate osqueryd logs on SIGHUP

Closes #1256

Note: Sometimes the test fails to rotate the log on SIGHUP, although
that doesn't appear to be the case with a long running process.
After some discussion and debugging with @zwass we agreed to call
t.Log and come back to this issue at a later time.
2017-03-03 12:21:48 -05:00
Zachary Wasserman
71a384f34a Show built in labels first in targets selector (#1338) 2017-03-02 15:40:50 -08:00
Zachary Wasserman
3d6ca7d5a4 Use sockjs to gracefully degrade websockets (#1255)
Use the [SockJS Protocol](https://github.com/sockjs/sockjs-protocol) to handle
bidirectional communication instead of plain websockets. This allows
distributed queries to function in situations in which they previously failed
(Load balancers not supporting websockets, issues with Safari and self-signed
certs, etc.).

Also includes fixes to the JS message handling logic where slightly different
message delivery semantics (when using XHR) were exposing bugs.

Fixes #1241, #1327.
2017-03-01 13:14:26 -08:00
Zachary Wasserman
8a0ae4ad9b Add migration to ensure 'All Hosts' is complete (#1330)
Due to recreating the 'All Hosts' label in #1282, we get inconsistent counts
for hosts that have not checked in since that migration. This seems acceptable
for other labels, but it is important that 'All Hosts' really includes all the
hosts.

This migration adds all the hosts into that label.

Fixes #1329
2017-03-01 12:49:02 -08:00
Mike Arpaia
f713c1fc08 Performance improvements for the packs API (#1331) 2017-03-01 12:56:13 -07:00
Zachary Wasserman
879363311c Allow deleted label names to be replaced (#1309)
Fixes #1310
2017-02-24 17:47:30 -08:00
Victor Vrantchan
897cb35e24 Allow checking in to license server when license is saved. (#1299)
* Allow checking in to license server when license is saved.

Closes #1290
Closes #1277
2017-02-24 19:33:42 -05:00
Zachary Wasserman
7564967813 Fix flapping host network interfaces (#1283)
Ensure that host network interfaces do not disappear when they (unexpectedly)
are returned with no updates from osquery. Add test to verify.

Fixes #1278
2017-02-24 14:37:47 -08:00
Mike Arpaia
184b268b10 Always return the All Hosts label (#1288)
close #1285
2017-02-23 17:41:12 -07:00
Zachary Wasserman
0b4b921f17 Re-add default decorators to osquery config (#1269)
These decorators were removed in #953 due to an osquery bug. That bug is now
fixed, and we are adding the decorators back. We also now use `load` decorators
rather than `interval` decorators because they seem to function more reliably.
2017-02-23 12:36:08 -08:00
Mike Arpaia
6c7862c42e Change label queries to use platform instead of build_platform (#1282) 2017-02-23 12:04:51 -07:00
Kyle Knight
91132e440b Update Password Reset Email (#1225) 2017-02-23 11:10:33 -05:00
Mike Arpaia
bce8985a38 Renaming license-checker to license (#1263) 2017-02-22 17:45:44 -07:00
Zachary Wasserman
5be9d69165 Ensure config values roundtrip properly through config_dump (#1266)
- Set the appropriate yaml tags for dumping
- Add test to verify roundtrip

Fixes #1261
2017-02-22 07:22:19 -08:00
Zachary Wasserman
44a95e3192 Retrieve correct platform in detail queries (#1265)
Previously we were using `build_platform`, which does not always properly
reflect the platform of the host running osquery. Now we should properly
retrieve the platform.

Fixes #1264
2017-02-21 19:22:34 -08:00
Zachary Wasserman
3bab6bae18 Use SELECT DISTINCT in favor of GROUP BY (#1251)
In some MySQL configurations, using a GROUP BY that doesn't refer to every
column in the SELECT will throw errors. Replace the use of GROUP BY with SELECT
DISTINCT as this is also more clear as to the intentions of the query.

Fixes #1249
2017-02-17 11:19:27 -08:00
Zachary Wasserman
c0d87cb69a Use persistent MySQL for local server, tmpfs MySQL for tests (#1245) 2017-02-17 09:15:10 -08:00
Victor Vrantchan
1ee94d4f75 add mysql client certificate support (#1240) 2017-02-16 17:14:00 -07:00
John Murphy
146ee18c62 Fix for bug #1236 (#1244) 2017-02-17 01:02:49 +08:00
John Murphy
f8d9f4095c Bug #1233 (#1242) 2017-02-16 23:01:20 +08:00
John Murphy
c8d284fd3c Email confirmation fix (#1231)
* Email confirmation fix

* Added test for incorrect auth user confirming e-mail change
2017-02-16 21:07:20 +08:00
Mike Arpaia
63f0d21b47 Return a full label response when modifying a label (#1228)
close #1222
2017-02-14 15:28:34 -07:00
Mike Arpaia
a3589892c3 A simpler attempt at using the payload pattern for scheduled queries (#1210) 2017-02-13 14:31:22 -07:00
John Murphy
6dc56ad5f0 Add ability to modify labels (#1202) 2017-02-12 12:27:43 +08:00
Victor Vrantchan
ec1c3b3b94 reverse order of arguments in error message. (#1196)
Fixing bug introduced in #1195
2017-02-09 20:09:06 -05:00
Zachary Wasserman
430a267b57 Send query campaign totals before results (#1193)
Avoids a bug where the frontend shows "x of 0 Hosts Returning y Records" by
ensuring that the expected total is updated before any results are sent.
2017-02-09 16:12:13 -08:00
Victor Vrantchan
cbd5c9d250 add namedError in addition to baseError (#1195) 2017-02-09 17:27:09 -05:00
Victor Vrantchan
5ddf788052 return a licensing error(instead of 500) when the public key does not match. (#1194) 2017-02-09 16:28:06 -05:00
John Murphy
7988076799 Licensed endpoints (#1188) 2017-02-10 02:43:45 +08:00
John Murphy
e9c4760979 License revocation checker (#1170) 2017-02-07 21:49:01 +08:00
John Murphy
9359604c13 Added GET licensing endpoint implementation (#1168)
* Added missing endpoint implementation
2017-02-04 00:41:17 +08:00
John Murphy
a1a0c91876 Licensing (#1123) 2017-02-03 04:30:59 +08:00
Victor Vrantchan
598b5c7cb8 replace soft deleted queries in NewQuery (#1159)
Uses the same strategy as #1002, #1016 to replace queries.

Closes #1148
2017-02-02 15:18:44 -05:00
Mike Arpaia
a000751bfe renaming kolide-ose to kolide (#1143) 2017-02-01 10:20:50 -07:00
Zachary Wasserman
7212d4c333 Add inline flag documentation in --help (#1135)
- Cleanup unused app.web_address flag
2017-01-31 10:06:30 -08:00
Victor Vrantchan
598e70ac55 always generate random enroll secret on setup (#1129)
Updated now that the UI has the ability to manage the enroll secret.
2017-01-30 12:48:43 -05:00
John Murphy
9465434826 Allow user to change email with confirmation (#1102)
* Change email functionality

* Code review changes for @groob

* Name change per @groob

* Code review changes per @marpaia

Also added addition non-happy path tests to satisfy concerns by @groob
2017-01-27 21:35:58 +08:00
Victor Vrantchan
ac95b764eb add host_hostname field to distributed query results row (#1081)
Add a hostname field to every distributed query result row in websocket response. By doing this calculation on the server, we're hoping to improve the rendering on the frontend client, which currently does the same calculation. 
 
Closes #1079
2017-01-23 12:37:03 -05:00
John Murphy
1a567b3f9f Labels response will return json with empty host_id rather than omit (#1078) 2017-01-23 23:37:09 +08:00
Mike Stone
378bf5571b connect new host enrollment (#1072) 2017-01-20 20:16:00 -05:00
Victor Vrantchan
54408ff9e4 move osquery enroll secret to appconfig (#1004)
For #995
2017-01-20 14:48:54 -05:00
Victor Vrantchan
60062834c8 add endpoint to serve the kolide certificate back to the user (#1025)
add endpoint to serve the kolide certificate back to the user

The API will attempt to establish a TLS connection and fetch the certificate from the TLS ConnectionState. 
The PEM encoded certificate will be served to the client in a JSON response as a base64 encoded string. 

Closes #1012
2017-01-20 14:32:10 -05:00
John Murphy
f2e7800f13 Send status message to browser to indicate query is done (#1020)
* Send status message to browser to indicate query is done

* Changed so only "finished" status message is emitted per @marpaia
2017-01-21 02:57:41 +08:00
John Murphy
7e136a018f removing trailing dots from osversion (#1066)
* removing trailing dots from osversion

* removed space from cutlist
2017-01-21 02:18:39 +08:00
John Murphy
92b93a3c3b Completely delete host (#1063) 2017-01-20 12:22:33 -05:00
Victor Vrantchan
eac718e937 add new status to host summary endpoint (#1057)
new_count shows all hosts that have been added to kolide in the last
24 hours
2017-01-20 08:57:47 -05:00
Victor Vrantchan
5f6f9388cd Account for hosts in labels in the total host count (#1056)
For #1039
2017-01-20 08:57:02 -05:00
Victor Vrantchan
02a6fe6870 increase query description length (#1055)
Closes #1053
2017-01-20 08:56:41 -05:00
John Murphy
28130b529b Addresses detail queries that don't return rows (#1044)
* If a detail query doesn't return results, log problem but don't interrupt additional query processing

* changed arg to ingest func to logger interface
2017-01-20 07:28:33 +08:00
Victor Vrantchan
4a095d09a8 default to username when query author doesn't have full name (#1037) 2017-01-19 16:46:28 -05:00
Victor Vrantchan
650bae2ffb remove dev mode from cli options (#1027) 2017-01-19 09:42:45 -05:00
Victor Vrantchan
c13b8cc0cf support JSON output for kolide logs (#1026) 2017-01-19 09:41:28 -05:00
Victor Vrantchan
d45c0103d1 Pack bugs (#1016)
* correctly list packs in response

Using append was adding a default pack response to the list of packs

* handle unique index for packs that exist but are deleted
2017-01-18 14:13:41 -05:00
Victor Vrantchan
42ed377312 when a duplicate email invite is detected, replace the old one (#1002)
* when a duplicate email invite is detected, replace the old one

* update sql
2017-01-18 10:40:51 -05:00
Victor Vrantchan
fa39dcd555 fix various issues with SMTP configuration (#1001)
* fix various issues with SMTP configuration

* handle SplitHostPort error
2017-01-18 10:05:09 -05:00
Kyle Knight
a6b45322ea Style Invite User email template (#998) 2017-01-18 09:43:09 -05:00
Victor Vrantchan
0c93c4db67 fallback to username if Name field is not set (#996)
* fallback to username if Name field is not set

* add orgname field to invite emails
2017-01-17 16:37:00 -05:00
John Murphy
41517aa7d9 Log in user automatically after successful setup (#993)
* Log in user automatically after successful setup

* Handle login failure by omitting token from response
2017-01-18 04:24:13 +08:00
Victor Vrantchan
1be80113d2 fix incorrect scoping in loop (#984)
The inner loop would only execute once and terminate, causing hosts
to show up with null network interfaces.

Closes #981
2017-01-17 12:02:22 -05:00
John Murphy
56b8772f13 Modify User (as a regular user) fails #891 (#959) 2017-01-18 00:43:59 +08:00
Victor Vrantchan
25c41cda94 add default targets for search results (#979)
Adds 5 most recently seen hosts + labels to the search targets response if the query is an empty string. 

Closes #921
2017-01-17 09:51:04 -05:00
Victor Vrantchan
6135f90f57 change recordlabelexecutions to use map[uint]bool instead of map[string]bool (#965)
Closes #345
2017-01-17 01:03:51 -05:00
John Murphy
a2fded1b2c Added hosts for each label (#963) 2017-01-17 06:57:05 +08:00
Mike Arpaia
5cc5220e5b Enforce uniqueness on query name (#915)
* Enforce uniqueness on query name

close #914

* catching the already exists error in MySQL
2017-01-16 15:20:15 -07:00
Mike Arpaia
066ec298b5 Don't return an error if statistics are requested and no hosts are available (#970)
close #966
2017-01-16 12:52:03 -07:00
John Murphy
f9fa65bfaa Changed query columns to TEXT type (#969) 2017-01-17 03:20:11 +08:00
Victor Vrantchan
f74216761a remove binaryFilesystem wrapper from bindata files (#964) 2017-01-16 10:16:50 -05:00
Victor Vrantchan
52a932bc6b Validate password requirements (#962)
Add validation for user password creation/reset
 - at least 7 chars
 - 1 number
 - 1 symbol 

consolidated service errors to a single file.
2017-01-15 18:23:09 -05:00
Victor Vrantchan
3802f3098e remove deprecated errors package (#961) 2017-01-14 15:00:09 -05:00
Mike Arpaia
581000bb64 Revert "Add some basic decorators to the osquery config (#875)" (#953)
This reverts commit 1d029073e5.
2017-01-13 13:21:54 -07:00
John Murphy
6f4dcdd082 Import Config - /config/import #366 (#764) 2017-01-13 12:35:25 -06:00
Victor Vrantchan
de3794b17b remove SMTP CLI flags. SMTP is now handles as app config in db (#949) 2017-01-13 08:54:16 -05:00
Victor Vrantchan
36dfad37ea allow osqueryd endpoints to enroll before app setup is complete (#931)
Closes #929
2017-01-11 19:40:58 -05:00
Mike Arpaia
dcfbe1b2d3 Ensure that manually attached hosts get packs (#922)
Previously, when determining which packs a host should get when it checked in, we were iterating each pack and only checking whether or not the host was apart of a label which was a target of the pack, but we were never checking whether or not the host had been added as a specific target of that pack. This PR makes the necessary modification to `svc.ListPacksForHost`.
2017-01-11 15:24:32 -07:00
Mike Arpaia
cf805aa66c Only return hosts which have been explicitly scheduled from packs API (#909)
* Only return hosts which have been explicitly scheduled from packs API

close #903

* better error handling

* documentation
2017-01-11 13:33:30 -07:00
Victor Vrantchan
1551e46fed add middleware to redirect setup to login if the app has an admin (#900)
user.
2017-01-11 14:05:07 -05:00
Zachary Wasserman
671255b105 Fix bug in saving host detail update time (#888)
Saving a new detail update time when the host details were not actually updated
caused detail updates to be missed. This PR fixes the existing test to catch
the bug, and fixes the bug.
2017-01-11 11:48:24 -07:00
Mike Arpaia
c2084026d1 Call MarkHostSeen from authenticate host (#881) 2017-01-11 09:30:13 -07:00
Mike Arpaia
9987983cb9 Simplifying SMTP Logic (#892)
* Simplifying SMTP Logic

This commit breaks the test email sending into it's own service method
(thus removing the capability from the API- if we want it back, we can
wire up another endpoint for just that). Additionally, error wrapping is
used through the new ModifyAppConfig service method to ensure that an
error or failed email will always result in an error while ensuring that
the submitted record always get committed (unless a serious error
happens).

* never wrap a nil error

* use err instead of individual errors
2017-01-11 01:27:09 -07:00
Victor Vrantchan
f276ff3d90 Wrap email errs (#884)
* wrap errors returned by SMTP client

* remove client hello
2017-01-10 23:56:32 -05:00