empayre/fleet
14
0
Fork 0
mirror of https://github.com/empayre/fleet.git synced 2024-11-06 17:05:18 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
436733763a
fleet/infrastructure/sandbox/Data
History
Zachary Winnerman 70524b6e3a
Sandbox data (#8523) 
* Initial work on sandbox data pipeline

* Update python code to have preliminary code

* Added in final code for sandbox-data.
Huge delay because I needed to double check licensing
2022-11-01 12:41:13 -04:00
..
lambda Sandbox data (#8523) 2022-11-01 12:41:13 -04:00
.gitignore Sandbox data (#8523) 2022-11-01 12:41:13 -04:00
main.tf Sandbox data (#8523) 2022-11-01 12:41:13 -04:00
README.md Sandbox data (#8523) 2022-11-01 12:41:13 -04:00
variables.tf Sandbox data (#8523) 2022-11-01 12:41:13 -04:00

README.md

The data pipeline

The data pipeline takes data from S3 using S3 notifications, filters for only the successful requests, then enriches the data with geoip data, then pipes it to kinesis. From kinesis, we stream the data to an Elasticsearch cluster for now, but this design allows for expansion into Salesforce and Mixpanel later on.