Copy-edit and Markdown lint (just nits) remaining developer docs

docs/wiki/development/osquery-sdk.md

@@ -1,14 +1,16 @@
+# The osquery SDK
+
 The osquery "public API" or SDK is the set of osquery headers and a subset of the source "cpp" files implementing what we call osquery **core**. The core code can be thought of as the framework or platform, it is everything except for the SQLite code and most table implementations. The public headers can be found in [osquery/include/osquery/](https://github.com/osquery/osquery/tree/master/include/osquery).
 
 osquery is organized into a **core**, **additional**, and **testing** during a default build from source. We call the set of public headers implementing **core** the 'osquery SDK'. This SDK can be used to build osquery outside of our CMake build system with a minimum set of dependencies. This organization better isolates OS API dependencies from development tools and libraries and provides a logical separation between code needed for extensions and module compiling.
 
-The public API and SDK headers are documented via **doxygen**. To generate web-based documentation, you will need to install doxygen, run `make docs` from the repository root, then open *./build/docs/html/index.html*.
+The public API and SDK headers are documented via **doxygen**. To generate web-based documentation, you will need to install doxygen, run `make docs` from the repository root, then open `./build/docs/html/index.html`.
 
 ## Extensions
 
-Extensions are separate processes that communicate over a Thrift IPC channel to osquery **core** in order to register one or more plugins or virtual tables. An extension allows you to develop independently: it may be compiled and linked using an external build system, against proprietary code. Your extension will be version-compatible with our publicly-built binary packages on [https://osquery.io/downloads](https://osquery.io/downloads).
+Extensions are separate processes that communicate over a [Thrift](https://thrift.apache.org/) IPC channel to osquery **core** in order to register one or more plugins or virtual tables. An extension allows you to develop independently: it may be compiled and linked using an external build system, against proprietary code. Your extension will be version-compatible with our publicly-built binary packages on [https://osquery.io/downloads](https://osquery.io/downloads).
 
-Extensions use osquery's [Thrift API](https://github.com/osquery/osquery/blob/master/osquery/extensions/thrift/osquery.thrift) to communicate between **osqueryi** or **osqueryd** and the extension process. Extensions are commonly written in C++, but can also be developed [in Python](https://github.com/osquery/osquery-python), [in Go](https://github.com/kolide/osquery-go), or in really any language that supports [Thrift](https://thrift.apache.org/).
+Extensions use osquery's [Thrift API](https://github.com/osquery/osquery/blob/master/osquery/extensions/thrift/osquery.thrift) to communicate between `osqueryi` or `osqueryd` and the extension process. Extensions are commonly written in C++, but can also be developed [in Python](https://github.com/osquery/osquery-python), [in Go](https://github.com/kolide/osquery-go), or in really any language that supports [Thrift](https://thrift.apache.org/).
 
 Only the osquery SDK provides the simple `startExtension` symbol that manages the life of your process, including the Thrift service threads and a watchdog.
 
@@ -61,7 +63,7 @@ int main(int argc, char* argv[]) {
 }
 ```
 
-The **osqueryi** or **osqueryd** processes start an "extension manager" Thrift service thread that listens for extension register calls on a UNIX domain socket. Extensions may only communicate if the processes can read/write to this socket. An extension process running as a non-privileged user cannot register plugins to an **osqueryd** process running as root. Both the osquery core and C++ extensions using `startExtension` will deregister plugins if the communication becomes latent. Both of these settings are configurable using gflags or the osquery config options.
+The `osqueryi` or `osqueryd` processes start an "extension manager" Thrift service thread that listens for extension register calls on a UNIX domain socket. Extensions may only communicate if the processes can read/write to this socket. An extension process running as a non-privileged user cannot register plugins to an `osqueryd` process running as root. Both the osquery core and C++ extensions using `startExtension` will deregister plugins if the communication becomes latent. Both of these settings are configurable using gflags or the osquery config options.
 
 ### Using the example extension
 
@@ -69,11 +71,11 @@ Please see the deployment [guide on extensions](../deployment/extensions.md) for
 
 If you [build from source](../development/building.md), you will build an example extension. The code can be found in the [`osquery/examples`](https://github.com/osquery/osquery/blob/master/osquery/examples/example_extension.cpp) folder; it adds a config plugin called "example" and additional table called "example". There are two ways to run an extension: load the extension at an arbitrary time after shell or daemon execution, or request an "autoload" of extensions. The auto-loading method has several advantages, such as allowing dependencies on external config plugins and inheriting the same process monitoring as is applied to the osquery core worker processes.
 
-The **osqueryi** shell also allows a quick and easy command-line autoload using `--extension`. Let's review both options.
+The `osqueryi` shell also allows a quick and easy command-line autoload using `--extension`. Let's review both options.
 
 First, to load the example extension in the osquery interactive shell, we start osqueryi:
 
-```
+```sh
 $ ./build/darwin/osquery/osqueryi
 osquery> SELECT path FROM osquery_extensions;
 +-------------------------------------+
@@ -87,7 +89,7 @@ osquery> ^Z
 
 Here we have started the osquery shell process, inspected the UNIX domain socket path it uses to communicate with extension processes, and then suspended the process temporarily.
 
-```
+```sh
 $ ./build/darwin/osquery/example_extension.ext --help
 osquery 1.7.0, your OS as a high-performance relational database
 Usage: example_extension.ext [OPTION]...
@@ -104,7 +106,7 @@ $ ./build/darwin/osquery/example_extension.ext --socket /Users/USERNAME/.osquery
 
 Before executing the extension we've inspected the potential CLI flags, which are a subset of the shell or daemon's [CLI flags](../installation/cli-flags.md). We executed the example extension in the background, so now we can resume the osqeury shell and use the 'example' table provided by the extension.
 
-```
+```sh
 [2] 98795
 $ fg
 [1]  - 98777 continued  ./build/darwin/osquery/osqueryi
@@ -121,8 +123,8 @@ If the responsible osquery shell or daemon process ends, the extension will dete
 
 The second, and simpler, way to manually load an extension is at the osqueryi command line with `--extension`:
 
-```
-$ ./build/darwin/osquery/osqueryi --extension ./build/darwin/osquery/example_extension.ext
+```sh
+./build/darwin/osquery/osqueryi --extension ./build/darwin/osquery/example_extension.ext
 ```
 
 ### Building external extensions

docs/wiki/development/pubsub-framework.md

@@ -1,8 +1,10 @@
+# The pub-sub evented data framework of osquery
+
 Most of osquery's virtual tables are generated when an SQL statement requests data. For example, the [time](https://github.com/osquery/osquery/blob/master/osquery/tables/utility/time.cpp) gets the current time and returns it as a single row. So whenever a call selects data from time, e.g., `SELECT * FROM time;` the current time of the call will return.
 
 From an operating systems perspective, query-time synchronous data retrieval is lossy. Consider the [processes](https://github.com/osquery/osquery/blob/master/osquery/tables/system/linux/processes.cpp) table: if a process like `ps` runs for a fraction of a moment there's no way `SELECT * FROM processes;` will ever include the details.
 
-To solve for this osquery exposes a [pubsub framework](https://github.com/osquery/osquery/tree/master/osquery/events) for aggregating operating system information asynchronously at event time, storing related event details in the osquery backing store, and performing a lookup to report stored rows query time. This reporting pipeline is much more complicated than typical query-time virtual table generation. The time of event, storage history, and applicable (final) virtual table data information must be carefully considered. As events occur, the rows returned by a query will compound, as such selecting from an event-based virtual table generator should always include a time range.
+To solve this, osquery exposes a [pubsub framework](https://github.com/osquery/osquery/tree/master/osquery/events) for aggregating operating system information asynchronously at event time, storing related event details in the osquery backing store, and performing a lookup to report stored rows query time. This reporting pipeline is much more complicated than typical query-time virtual table generation. The time of event, storage history, and applicable (final) virtual table data information must be carefully considered. As events occur, the rows returned by a query will compound, as such selecting from an event-based virtual table generator should always include a time range.
 
 If no time range is provided, as in: `SELECT * FROM process_events;`, it is assumed you want to scan from `t=[0, now)`. Otherwise, all of the `*_events` tables must have a `time` column, this is used to optimize searching: `SELECT * FROM process_events WHERE time > NOW() - 300;`.
 
@@ -10,7 +12,7 @@ If no time range is provided, as in: `SELECT * FROM process_events;`, it is assu
 
 Almost every pubsub-based table ends with a `_events` or `_changes`. These tables will perform lookups into the osquery backing storage: RocksDB, for events buffered by the subscribers. These tables are a "query-time" abstraction that allow you to use SQL aggregations and a `time` column for optimizing lookups.
 
-When using the `osqueryi` shell, these tables will mostly remain empty. This is because the event loops start and stop with the process. If the shell is not running, no events are being buffered. Furthermore, some of the APIs used by the runloops require super user privileges or non-default flags and options. The shell does NOT communicate with the osquery daemon, nor does it use the same RocksDB storage. Thus the shell cannot be used to explore events buffered by the daemon.
+When using the `osqueryi` shell, these tables will mostly remain empty. This is because the event loops start and stop with the process. If the shell is not running, no events are being buffered. Furthermore, some of the APIs used by the runloops require super-user privileges or non-default flags and options. The shell does **not** communicate with the osquery daemon, nor does it use the same RocksDB storage. Thus the shell cannot be used to explore events buffered by the daemon.
 
 The buffered events will eventually expire! The `--events_expiry` flag controls the lifetime of buffered events. This is set to 1 day by default, this expiration occurs when events are selected from their subscriber table. For example: the `process_events` subscriber will buffer process starts until a query selects from this table. At that point all results will be returned and immediately after, any event that happened `time-86400` seconds ago will be deleted. If you select from this table every second you will constantly see a window of 1 day's worth of process events.
 
@@ -24,13 +26,13 @@ The pubsub runflow is exposed as a publisher `setUp()`, a series of `addSubscrip
 
 ## Example: inotify
 
-Filesystem events are the simplest example, let's consider Linux's inotify framework. [osquery/events/linux/inotify.cpp](https://github.com/osquery/osquery/blob/master/osquery/events/linux/inotify.cpp) is exposed as an osquery publisher.
+Filesystem events are the simplest example. Let's consider Linux's inotify framework: [osquery/events/linux/inotify.cpp](https://github.com/osquery/osquery/blob/master/osquery/events/linux/inotify.cpp) implements an osquery publisher.
 
-There's an array of yet-to-be-implemented uses of the inotify publisher, but a simple example includes querying for every change to "/etc/passwd". The [osquery/tables/events/linux/file_events.cpp](https://github.com/osquery/osquery/blob/master/osquery/tables/events/linux/file_events.cpp) table uses a pubsub subscription and implements a subscriber. The subscriptions are constructed from the configuration. See the file [integrity monitoring deployment](../deployment/file-integrity-monitoring.md) guide for details.
+There's a list of yet-to-be-implemented uses of the inotify publisher, but a simple example includes querying for every change to `/etc/passwd`. The [osquery/tables/events/linux/file_events.cpp](https://github.com/osquery/osquery/blob/master/osquery/tables/events/linux/file_events.cpp) table uses a pubsub subscription and implements a subscriber. The subscriptions are constructed from the configuration. See the file [integrity monitoring deployment](../deployment/file-integrity-monitoring.md) guide for details.
 
 ## Event Subscribers
 
-Let's continue to use the inotify event publisher as an example. And let's implement a table that reports new files created in "/etc/". The first thing we need is a [table spec](creating-tables.md):
+Let's continue to use the inotify event publisher as an example. And let's implement a table that reports new files created in `/etc/`. The first thing we need is a [table spec](creating-tables.md):
 
 ```python
 table_name("new_etc_files")

docs/wiki/development/unit-tests.md

@@ -1,8 +1,10 @@
+# Unit testing in osquery
+
 All commits to osquery should be well unit-tested. Having tests is useful for many reasons. In addition to the subtle advantage of being able to assert program correctness, tests are often the smallest possible executable which can run a given bit of code. This makes testing new features for memory leaks much easier. Using tools like valgrind in conjunction with compiled tests, we can directly analyze the desired code with minimal outside influence.
 
 ## Writing a test
 
-**Prerequisite**
+### Prerequisite
 
 This guide is going to take you through the process of creating and building a new unit test in the osquery project.
 
@@ -10,7 +12,7 @@ First ensure that you can properly build the code, by referring to the ["buildin
 
 Before you modify osquery code (or any code for that matter), make sure that you can successfully execute all tests. The steps for building and running tests are particular to the platform and build toolchain you are using, so again refer to the ["building osquery"](building.md) guide for the appropriate information for your setup.
 
-**Adding a test**
+## Adding a test
 
 We'll create a test in the "osquery/examples" subdirectory of the main repository. Let's create a file "example_test.cpp" in that directory.
 
@@ -48,6 +50,6 @@ ADD_OSQUERY_TEST(example_test example_test.cpp)
 
 After you specify the test sources, add whatever libraries you have to link against and properly set the compiler flags, make sure you call `ADD_TEST` with your unit test. This registers it with CTest (CMake's test runner).
 
-**Extending the test**
+## Extending a test
 
 Your test is just C++ code. Use the [Google Test documentation](https://github.com/google/googletest/blob/master/googletest/docs/primer.md#assertions) to assist you in writing meaningful tests.