fleet/server/service
Matteo Piano c89cd370d5
Add AWS S3 as file carving backend (#126)
This adds the option to set up an S3 bucket as the storage backend for file carving (partially solving #111).

It works by using the multipart upload capabilities of S3 to maintain compatibility with the "upload in blocks" protocol that osquery uses. It does this basically replacing the carve_blocks table while still maintaining the metadata in the original place (it would probably be possible to rely completely on S3 by using object tagging at the cost of listing performance). To make this pluggable, I created a new field in the service struct dedicated to the CarveStore which, if no configuration for S3 is set up will be just a reference to the standard datastore, otherwise it will point to the S3 one (effectively this separation will allow in the future to add more backends).
2020-12-16 09:16:55 -08:00
..
testdata add endpoint to serve the kolide certificate back to the user (#1025) 2017-01-20 14:32:10 -05:00
client_appconfig.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
client_carves.go Add AWS S3 as file carving backend (#126) 2020-12-16 09:16:55 -08:00
client_debug.go Add debug endpoints and associated fleetctl commands (#45) 2020-11-17 17:12:37 -08:00
client_errors.go Improve client error messages with unexpected server errors (#1776) 2018-05-09 16:54:23 -07:00
client_hosts.go Add file carving support (#15) 2020-11-04 20:45:16 -08:00
client_labels.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
client_live_query.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
client_options.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
client_packs.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
client_queries.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
client_sessions.go Add file carving support (#15) 2020-11-04 20:45:16 -08:00
client_setup.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
client_targets.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
client_users.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
client.go Fix fleetctl Windows issues (#40) 2020-11-17 16:02:14 -08:00
debug_handler_test.go Add debug endpoints and associated fleetctl commands (#45) 2020-11-17 17:12:37 -08:00
debug_handler.go Add debug endpoints and associated fleetctl commands (#45) 2020-11-17 17:12:37 -08:00
endpoint_appconfig_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_appconfig.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_campaigns.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_carves.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_certificate.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_change_email.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_hosts.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_invites.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_labels.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_middleware_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_middleware.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_osquery_options.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_osquery.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_packs.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_queries.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_scheduled_queries.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_sessions.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_setup.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_status.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_targets.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
endpoint_users_test.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
endpoint_users.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
frontend.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
handler_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
handler.go Minor code/comment cleanups (#47) 2020-11-18 11:10:55 -08:00
http_auth_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
http_auth.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_appconfig.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_campaigns.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_email_change.go Reduce verbosity of default logging (#2225) 2020-04-22 14:01:34 -07:00
logging_hosts.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_invites.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_labels.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_osquery_options.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_osquery.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_packs.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_queries.go Fix logging for query creation on error (#132) 2020-12-14 18:45:03 -08:00
logging_scheduled_queries.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_sessions.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging_users.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
logging.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
metrics_appconfig.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
metrics_change_email.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
metrics_invites.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
metrics_labels.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
metrics_sessions.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
metrics_users.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
metrics.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_appconfig_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_appconfig.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_campaigns.go Prevent SQLite ATTACH from being used in queries (#129) 2020-12-14 18:13:34 -08:00
service_carves_test.go Add AWS S3 as file carving backend (#126) 2020-12-16 09:16:55 -08:00
service_carves.go Add AWS S3 as file carving backend (#126) 2020-12-16 09:16:55 -08:00
service_certificate_test.go add endpoint to serve the kolide certificate back to the user (#1025) 2017-01-20 14:32:10 -05:00
service_certificate.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
service_errors.go removing license code (#1551) 2017-09-01 10:42:46 -06:00
service_hosts_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_hosts.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_invites_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_invites.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_labels_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_labels.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_osquery_options.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_osquery_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_osquery.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_packs_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_packs.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_queries_test.go Prevent SQLite ATTACH from being used in queries (#129) 2020-12-14 18:13:34 -08:00
service_queries.go Prevent SQLite ATTACH from being used in queries (#129) 2020-12-14 18:13:34 -08:00
service_scheduled_queries_test.go Allow scheduling same query more than once in UI (#96) 2020-12-04 09:50:39 -08:00
service_scheduled_queries.go Allow scheduling same query more than once in UI (#96) 2020-12-04 09:50:39 -08:00
service_sessions_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_sessions.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_status.go Add ability to disable live queries (#2167) 2020-01-13 16:53:04 -08:00
service_targets_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_targets.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service_users_test.go Allow @ character in usernames (#37) 2020-11-13 15:52:20 -08:00
service_users.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
service.go Add AWS S3 as file carving backend (#126) 2020-12-16 09:16:55 -08:00
transport_appconfig.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
transport_campaigns.go Add fleetctl query command (#1784) 2018-05-17 15:54:34 -07:00
transport_carves.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
transport_change_email.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
transport_error.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
transport_hosts.go Add host additional info filters (#28) 2020-11-13 16:33:25 -08:00
transport_invites_test.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
transport_invites.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
transport_labels_test.go Fix labels UI issues in Fleet 2.0 (#1830) 2018-06-18 10:09:08 -07:00
transport_labels.go Implement pagination of hosts in the web UI 2020-07-21 14:05:46 -07:00
transport_osquery_options.go Add endpoints for apply/get of each spec type (#1752) 2018-05-03 18:01:57 -07:00
transport_osquery_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
transport_osquery.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
transport_packs_test.go Fix pack and query UI issues in Fleet 2.0 (#1829) 2018-06-15 10:13:11 -04:00
transport_packs.go Fix pack and query UI issues in Fleet 2.0 (#1829) 2018-06-15 10:13:11 -04:00
transport_queries_test.go Delete spec entities by name instead of ID (#1755) 2018-05-04 11:05:55 -07:00
transport_queries.go Fix pack and query UI issues in Fleet 2.0 (#1829) 2018-06-15 10:13:11 -04:00
transport_scheduled_queries_test.go Allow scheduled query shard to be set to empty in frontend (#2093) 2019-08-13 09:42:02 -07:00
transport_scheduled_queries.go Fix pack and query UI issues in Fleet 2.0 (#1829) 2018-06-15 10:13:11 -04:00
transport_sessions_test.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
transport_sessions.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
transport_setup.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
transport_targets_test.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
transport_targets.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
transport_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
transport_users_test.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
transport_users.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
transport.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
util_test.go Add AWS S3 as file carving backend (#126) 2020-12-16 09:16:55 -08:00
validation_app_config_test.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
validation_app_config.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
validation_invites.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
validation_sessions.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
validation_setup.go Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
validation_users.go Allow @ character in usernames (#37) 2020-11-13 15:52:20 -08:00