valitydev/SigmaHQ
14
0
Fork 0
mirror of https://github.com/valitydev/SigmaHQ.git synced 2024-11-08 10:13:57 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,865 Commits 20 Branches 25 Tags 21 MiB
01bd5cf0e0
Commit Graph

326 Commits

Author SHA1 Message Date
Thomas Patzke
265ce115a0 Fixed conditional field mapping usage in mapping chains 2018-10-16 13:57:51 +02:00
Michael H
5b33713ef8 Quick fix for string formatting bug 2018-10-13 20:21:37 -05:00
Michael H
38ec257f7e Re-doing LogName formatting 2018-10-13 20:18:57 -05:00
Michael H
9f48265eb1 Adding re.sub for LogName that accounts for expression grouping 2018-10-13 20:09:54 -05:00
Michael H
aabaa0257b Merge branch 'master' of https://github.com/Neo23x0/sigma 2018-10-06 20:12:15 -05:00
Michael H
4b85a34b34 Added CSV option to powershell backend 2018-10-06 20:08:20 -05:00
Thomas Patzke
e28bc35cad Apply field mappings in generation of log source condition 2018-10-06 23:38:35 +02:00
Daniel Roethlisberger
fc45df144c Improve the comments on the optimizer 2018-10-03 13:44:03 +02:00
Daniel Roethlisberger
87aa1b5521 Move optimizer to sigma.parser.condition to enable it for all backends 2018-10-03 00:24:31 +02:00
Daniel Roethlisberger
cd3661b60c Fix optimization of NOT corner cases 2018-10-02 22:48:33 +02:00
Daniel Roethlisberger
bed88cf813 Make uniq work for lists within definitions 2018-10-02 22:12:54 +02:00
Daniel Roethlisberger
7165128fa5 Remove None from AST - fixes None-related test failures 2018-10-02 21:44:37 +02:00
Daniel Roethlisberger
2242fc5ac8 Optimize the boolean expressions in the AST before generating output 
Add code optimizing the boolean expressions in the abstract syntax tree
before generating output using the backend.

The main idea behind optimizing the AST is that less repeated terms is
generally better for backend performance.  This is especially relevant
to backends that do not perform any query language optimization down
the road, such as those that generate code.

The following optimizations are currently performed:

-   Removal of empty OR(), AND()
-   OR(X), AND(X)                 =>  X
-   OR(X, X, ...), AND(X, X, ...) =>  OR(X, ...), AND(X, ...)
-   OR(X, OR(Y))                  =>  OR(X, Y)
-   OR(AND(X, ...), AND(X, ...))  =>  AND(X, OR(AND(...), AND(...)))
-   NOT(NOT(X))                   =>  X

A common example for when these suboptimal rules actually occur in
practice is when a rule has multiple alternative detections that are
OR'ed together in the condition, and all of the detections include a
common element, such as the same EventID.

This implementation is not optimized for performance and will perform
poorly on very large expressions.
 2018-10-02 21:14:25 +02:00
Karneades
468af42de5 Add missing event id list handling in PowerShell backend 2018-09-29 14:43:28 +02:00
Karneades
c289484c5c Improve default field handling in PowerShell backend 2018-09-29 12:29:44 +02:00
Karneades
c66b00356d Add initial version of PowerShell backend 
* Add PowerShell backend
* Add PowerShell config file

State: Work in progress :)

See https://github.com/Neo23x0/sigma/issues/94
 2018-09-23 21:41:48 +02:00
Thomas Patzke
2fbf17ff34 Addition and resolution of field mapping chains explicitely checks for list 2018-09-13 16:22:29 +02:00
Thomas Patzke
41a8ef2fd9 Implemented resolve_fieldname in FieldMappingChain 2018-09-13 14:56:31 +02:00
Thomas Patzke
2330306db1 Added merged field mapping and log sources dict to config chain 2018-09-13 14:55:05 +02:00
Thomas Patzke
ba76f04fe6 Merging of raw configurations in configuration chains 2018-09-13 13:49:36 +02:00
Thomas Patzke
d81946df39 Stacked configurations 
- Added log source rewriting
- Removed log source merging condition type setting
- Simplified SigmaLogsourceConfiguration constructor
- Condition is generated in SigmaParser instead of SigmaLogsourceConfiguration

Missing:
- Merging of raw config dict for backends that rely on this (es-dsl)
 2018-09-12 23:40:22 +02:00
Thomas Patzke
210f7ac044 Rewrote logsource definition merging to set generator 2018-09-12 22:29:51 +02:00
Thomas Patzke
f3c60a6309 Added tag filtering to sigmac 2018-09-06 00:57:54 +02:00
Thomas Patzke
7f875af1ca Fixed WDATP backend 
It never generated any output due to missing return in generate()
method.
 2018-09-06 00:31:40 +02:00
Thomas Patzke
1d7722c1cb Added configuration and field mapping chains 
Missing: field name mapping of log source conditions.
 2018-08-27 00:17:27 +02:00
James Dickenson
29bed766dd removed re-introduced output class from qradar backend. fixed list handling error. 2018-08-21 22:45:12 -07:00
James Dickenson
468f040c0a Merge branch 'qradar-dev' 2018-08-20 21:54:30 -07:00
James Dickenson
9a61f40cef added support flor flow data in qradar backend 2018-08-16 21:44:17 -07:00
James Dickenson
a8d1831382 Added aggregation support for qradar backend 2018-08-13 23:04:10 -07:00
Thomas Patzke
dce4b4825d Fixed aggregations without field name 
Generated query contained field name "None".
 2018-08-10 15:07:07 +02:00
Thomas Patzke
e0b3f91b2a Removed empty line 2018-08-08 23:15:13 +02:00
Thomas Patzke
af9f636199 Removal of backend output classes 
Breaking change: Instead of feeding the output class with the results,
they are now returned as strings (*Backend.generate()) or list
(SigmaCollectionParser.generate()). Users of the library must now take
care of the output to the terminal, files or wherever Sigma rules should
be pushed to.
 2018-08-02 22:41:32 +02:00
Thomas Patzke
1c9d0a176e Moved const_start into class definition 2018-07-28 23:51:33 +02:00
Thomas Patzke
df74460629 Fixed imports after config split 2018-07-27 23:54:18 +02:00
Thomas Patzke
e02af9aa37 Merge config split branches 2018-07-27 23:16:50 +02:00
Thomas Patzke
eb440b3357 Split config - code removal from configuration 2018-07-27 23:02:35 +02:00
Thomas Patzke
36ada66007 Split config - Copy configuration 2018-07-27 23:01:41 +02:00
Thomas Patzke
920c4b061d Split config - code removal from filter 2018-07-27 22:35:30 +02:00
Thomas Patzke
d235a9e017 Split config - Copy filter 2018-07-27 00:23:22 +02:00
Thomas Patzke
50a6a92d20 Split config - code removal from exceptions 2018-07-27 00:17:35 +02:00
Thomas Patzke
405bc4a0d1 Split config - Copy exception 2018-07-27 00:17:13 +02:00
Thomas Patzke
096bc35447 Split config - code removal from mapping 2018-07-27 00:15:14 +02:00
Thomas Patzke
4ffbb25960 Split config - Copy mapping 2018-07-27 00:13:19 +02:00
Thomas Patzke
1c4c67053c Fixes for parser split 
* Fixed imports
* Rename
 2018-07-27 00:02:07 +02:00
Thomas Patzke
88a4a5d36a Merge parser split branches 2018-07-26 23:42:09 +02:00
Thomas Patzke
595327ace4 Split parser - code removal from condition 2018-07-26 23:40:22 +02:00
Thomas Patzke
c8043368bd Split parser - code removal from rule 2018-07-26 22:43:49 +02:00
Thomas Patzke
294ca20350 Split parser - code removal from collection 2018-07-26 22:28:33 +02:00
Thomas Patzke
3a0de01bad Split parser - code removal from base 2018-07-26 22:22:21 +02:00
Thomas Patzke
b9425d13df Split parser - code removal from exceptions 2018-07-26 22:18:21 +02:00
Thomas Patzke
e550bf5c3b Split parser - Copy base 2018-07-26 22:15:04 +02:00
Thomas Patzke
a2329de03c Split parser - Copy rule 2018-07-26 22:07:38 +02:00
Thomas Patzke
1abb13c5d9 Split parser - Copy condition 2018-07-24 00:13:37 +02:00
Thomas Patzke
a8501cb446 Split parser - Copy exceptions 2018-07-24 00:08:23 +02:00
Thomas Patzke
983ee6eeb9 Splitting parser - copying collections 2018-07-24 00:06:02 +02:00
Thomas Patzke
54f5870658 Removed debugging code 2018-07-24 00:04:24 +02:00
Thomas Patzke
b76fa884ec Changed copyright notices accordingly 2018-07-24 00:01:16 +02:00
Thomas Patzke
fbde251ebc Added missing exception import in ES backend 2018-07-22 09:26:25 +02:00
Thomas Patzke
91e6b8ca6b Merging refactoring changes into master 2018-07-22 09:23:07 +02:00
Thomas Patzke
cf175d7b7e Removal from sigma.backends.qradar 2018-07-22 09:14:50 +02:00
Thomas Patzke
097660c678 Splitting backends - Copy qradar.py 2018-07-22 09:12:29 +02:00
Thomas Patzke
c8e21b3f24 Fixing after split 
* Fixing imports
* Discovery in new sub modules
 2018-07-21 01:09:02 +02:00
Thomas Patzke
b85aec6157 Merging backend split branches 2018-07-21 00:59:50 +02:00
Thomas Patzke
3e2184ac61 Removal from sigma.backends.elasticsearch 2018-07-21 00:37:36 +02:00
nikotin
b5f27d75be Added Qradar backend 2018-07-17 15:25:06 +03:00
Thomas Patzke
c2b1a58813 Removal from sigma.backends.wdatp 2018-07-10 23:49:39 +02:00
Thomas Patzke
45782c6328 Removal from sigma.backends.splunk 2018-07-10 23:48:47 +02:00
Thomas Patzke
46f29d2eb2 Removal from sigma.backends.output 2018-07-10 23:47:41 +02:00
Thomas Patzke
2d4145cfe8 Removal from sigma.backends.discovery 2018-07-10 23:46:52 +02:00
Thomas Patzke
83acff6859 Splitting backends - Copy discovery.py 2018-07-10 23:46:16 +02:00
Thomas Patzke
d340487e94 Removal from sigma.backends.base 2018-07-10 23:44:14 +02:00
Thomas Patzke
2e7d366da5 Removal from sigma.backends.mixins 2018-07-10 23:42:38 +02:00
Thomas Patzke
bb78c1428e Removal from sigma.backends.logpoint 2018-07-10 23:41:15 +02:00
Thomas Patzke
2edeaee748 Removal from sigma.backends.graylog 2018-07-10 23:40:17 +02:00
Thomas Patzke
e5baca0ac4 Removal from sigma.backends.qualys 2018-07-10 23:39:18 +02:00
Thomas Patzke
fdfe346adc Removal from sigma.backends.exceptions 2018-07-10 23:37:59 +02:00
Thomas Patzke
7fbc3a35a3 Removal from sigma.backends.cli 2018-07-10 23:33:40 +02:00
Thomas Patzke
881f72e418 Removal from sigma.backends.tools 2018-07-10 23:32:42 +02:00
Thomas Patzke
09ac41949c Removal from sigma.backends.archsight 2018-07-10 23:22:36 +02:00
Thomas Patzke
04b89befce Splitting backends - Copy elasticsearch.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
bb9bef4deb Splitting backends - Copy wdatp.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
72480d304b Splitting backends - Copy splunk.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
c5d5c52850 Splitting backends - Copy output.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
0c93040da5 Splitting backends - Copy base.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
a8e19bb4ba Splitting backends - Copy mixins.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
116fe16512 Splitting backends - Copy logpoint.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
b621e9c3a8 Splitting backends - Copy graylog.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
a2ee36eac7 Splitting backends - Copy qualys.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
32c70b26d8 Splitting backends - Copy exceptions.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
43d951b173 Splitting backends - Copy cli.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
a6cd7a3d6b Splitting backends - Copy tools.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
7a2b1ae790 Splitting backends - Copy arcsight.py 2018-07-10 23:15:04 +02:00
Thomas Patzke
d064d24fbe Sigmac WDATP backend: renamed action types 2018-07-10 22:49:38 +02:00
Roey
14464f8c79 Added support of splunk dashboards (xml) 2018-06-22 14:17:58 +02:00
Thomas Patzke
7d1b801858 Merge branch 'devel-sigmac-wdatp' 2018-06-22 00:43:23 +02:00
Thomas Patzke
31727b3b25 Added Windows Defender ATP backend 
Missing:
* Aggregations
 2018-06-22 00:03:10 +02:00
Thomas Patzke
e72c0d5de4 SingleTextQueryBackend ignores empty components in composed queries 
Example: one component of a AND-composition is ignored if invoked
generate* call returns None.
 2018-06-21 23:59:41 +02:00
Thomas Patzke
d8a7bcad39 Reordered rule generation 
Generation of query parts before and after main query gives access to
information possibly gathered while main query generation.
 2018-06-21 23:50:13 +02:00
Thomas Patzke
8ddb369df3 Integration of Qualys backend 
* Changed description text to one-liner
* Output to intended class
* Minor code optimizations
 2018-06-07 23:31:09 +02:00
Thomas Patzke
ce9db548ff Integration of ArcSight backend 
* Rename
* Changed description to one line to beautify output of backend list
* Small bugfix in handling of numeric values
 2018-06-07 23:04:36 +02:00
nikotin
d13e8d7bd3 Added ArcSight & Qualys backends 2018-06-07 16:18:23 +03:00
Paul Dutot
715a88542d Graylog backend added 2018-05-17 15:51:25 +01:00
milkmix
37ee355a77 patched es-dsl 2018-05-17 08:44:50 +02:00
Thomas Patzke
738d03c751 Fixed position of line separation if rulecomment and verbose is active 2018-05-13 22:36:51 +02:00
Thomas Patzke
7647587a8b Fixed quoting of backslashes in generated queries 2018-05-01 00:45:59 +02:00
Thomas Patzke
de2ed08695 Merge branch 'ci-es' 2018-05-01 00:34:11 +02:00
Thomas Patzke
e411039b56 Fixed escaping of \u in Elasticsearch Query String queries 2018-05-01 00:05:16 +02:00
milkmix
0b3b0c3aaf imported es-dsl code from repo 2018-04-06 17:36:11 +02:00
Thomas Patzke
22ee6f4521 sigmac: escaped wildcards (\* and \?) are passed in generated query 2018-03-29 11:15:20 +02:00
Thomas Patzke
5f8b60cc24 sigmac: Improved fieldlist backend 
* Unique list of fields for multiple rules
* Aggregation support
 2018-03-22 00:03:51 +01:00
Thomas Patzke
0018503501 sigmac: Fixed rulecommend backend option 2018-03-21 01:13:10 +01:00
Thomas Patzke
4a9849b161 sigmac: improved backend options 
* parsing in main class
* help
 2018-03-21 00:53:44 +01:00
Thomas Patzke
bd20ffdad9 sigmac/kibana: curl URL quoted 2018-03-21 00:22:00 +01:00
Thomas Patzke
3f5f3a8d50 sigmac: Remove problematic characters from rule identifiers 2018-03-17 00:44:50 +01:00
Thomas Patzke
f6858c436a sigmac: Kibana curl output generates one index pattern line per pattern 2018-03-16 23:53:12 +01:00
Thomas Patzke
13ec4c3e3b sigmac: Kibana curl importer script 2018-03-11 00:25:12 +01:00
Thomas Patzke
7141729ffc sigma/parser: Introduced new conditions 
* Any definition: 1 of them
* All definitions: all of them
* Any of selected definitions: 1 of def* (wildcard)
* All of selected definitions: all of def* (wildcard)
 2018-03-06 23:13:42 +01:00
Thomas Patzke
647fc6187a sigmac: Added proper 'Content-Type' header for xpack-watcher backend 2018-03-04 22:58:15 +01:00
Thomas Patzke
89aa300bbc Improved xpack-watcher actions 
* Log and mail
* Details in message
 2018-02-09 00:03:41 +01:00
Thomas Patzke
8336929d76 XPack Watcher Backend: Improved aggregation capabilities 
* Aggregation with "...count(field)...", "...by field..." and
  combination of both
* Still only count() supported
 2018-02-08 22:17:35 +01:00
Thomas Patzke
4762a1cc30 Removed abandoned SigmaAggregationParser.trans_timeframe() method 2018-02-05 23:30:00 +01:00
Thomas Patzke
ec3f0f6d60 Fixed before/after logic 
If nothing was generated "None" was printed.
 2018-02-01 22:49:02 +01:00
Thomas Patzke
76bdcba71f Added rulecomment option to all single-query output backends 
Prints comment with rule before output.
 2018-01-27 23:48:10 +01:00
Thomas Patzke
f3d19f394e Fixed encoding issues 
Some OS environments don't use UTF-8 as default encoding. Enforced it
for output files and stdout.
 2017-12-13 00:12:56 +01:00
Thomas Patzke
09d40ab2da Finished packaging and refactoring 2017-12-08 22:32:39 +01:00
Thomas Patzke
68d8afe4e6 Intermediate refactoring commit: moving code into package 
Further splitting sigma.py into smaller parts.
 2017-12-08 21:45:05 +01:00