support for pluggable service registries, added consul kv support, etcd support, renamed to registrator

This commit is contained in:
Jeff Lindsay 2014-07-22 15:35:00 -05:00
parent ef5f6a0031
commit 00a5c07039
10 changed files with 476 additions and 253 deletions

2
.gitignore vendored
View File

@ -1,2 +1,2 @@
release
docksul
dockser

View File

@ -1,8 +1,8 @@
FROM progrium/busybox
MAINTAINER Jeff Lindsay <progrium@gmail.com
ADD ./stage/docksul /bin/docksul
ADD ./stage/dockser /bin/dockser
ENV DOCKER_HOST unix:///tmp/docker.sock
ENTRYPOINT ["/bin/docksul"]
ENTRYPOINT ["/bin/dockser"]

View File

@ -1,11 +1,11 @@
NAME=docksul
NAME=registrator
HARDWARE=$(shell uname -m)
VERSION=0.1.0
VERSION=0.2.0
build:
mkdir -p stage
go build -o stage/docksul
docker build -t docksul .
go build -o stage/dockser
docker build -t registrator .
release:
rm -rf release

187
README.md
View File

@ -1,113 +1,174 @@
# docksul
# Registrator
A Docker-Consul bridge that automatically registers containers with published ports as Consul services. As Docker containers are started, docksul will inspect them for published ports and register them as services with Consul. As containers stop, the services are deregistered. If the default service descriptions are unsuitable, you can customize them with environment variables on the container.
Service registry bridge for Docker
Although available standalone, docksul is used as a component of Consulate and it's recommended you use Consulate instead unless you know what you're doing.
Registrator listens for Docker events and register/deregisters services for containers based on published ports and metadata from the container environment. Registrator supports pluggable service registries, which currently includes Consul and Etcd.
## Starting docksul
By default, it can register services without any user-defined metadata. This means it works with *any* container, but allows the container author or Docker operator to override/customize the service definitions.
docksul assumes the default Docker socket at `file:///var/run/docker.sock` or you can override it with `DOCKER_HOST`. It also uses `0.0.0.0:8500` for Consul, but you can override it by passing an IP and port as an argument.
## Starting Registrator
$ docksul [consul-addr]
Registrator assumes the default Docker socket at `file:///var/run/docker.sock` or you can override it with `DOCKER_HOST`. The only argument is a registry URI, which is described in the next section.
You can run it as a container, but you must pass the Docker socket file as a mount to `/tmp/docker.sock`:
$ registrator <registry-uri>
$ docker run -d -v /var/run/docker.sock:/tmp/docker.sock progrium/docksul [consul-addr]
You can run it as a container, but you must pass the Docker socket file as a mount to `/tmp/docker.sock`, and it's a good idea to set the hostname to the machine host:
$ docker run -d \
-v /var/run/docker.sock:/tmp/docker.sock \
-h $HOSTNAME progrium/registrator <registry-uri>
### Registry URIs
The registry backend to use is defined by a URI. The scheme is the supported registry name, and an address. Registries based on key-value stores like etcd and Zookeeper (not yet supported) can specify a key path to use to prefix service definitions. Registries may also use query params for other options. See further down on adding support for other registries.
#### Consul Service Catalog (recommended)
To use the Consul service catalog, specify a Consul URI without a path. If no host is provided, `127.0.0.1:8500` is used. Examples:
$ registrator consul://10.0.0.1:8500
$ registrator consul:
#### Consul Key-value Store
The Consul backend also lets you just use the key-value store. This mode is enabled by specifying a path. Consul key-value support does not currently use service attributes/tags. Example URIs:
$ registrator consul:///path/to/services
$ registrator consul://192.168.1.100/services
Service definitions are stored as:
<registry-uri-path>/<service-name>/<service-id> = <ip>:<port>
#### Etcd Key-value Store
Etcd support works similar to Consul key-value. It also currently doesn't support service attributes/tags. If no host is provided, `127.0.0.1:4001` is used. Example URIs:
$ registrator etcd:///path/to/services
$ registrator etcd://192.168.1.100/services
Service definitions are stored as:
<registry-uri-path>/<service-name>/<service-id> = <ip>:<port>
## How it works
### One published port, the simple case
Services are registered and deregistered based on container start and die events from Docker. The service definitions are created with information from the container, including user-defined metadata in the container environment.
If a container publishes one port, one service will be created using the host port. By default, the service will be named after the base name of the image. For example:
For each published port of a container, a `Service` object is created and passed to the `ServiceRegistry` to register. A `Service` object looks like this and defaults explained in the comments:
type Service struct {
ID string // <hostname>:<container-name>:<internal-port>
Name string // <basename(container-image)>[-<internal-port> if >1 published ports]
Port int // <host-port>
IP string // <host-ip> || <resolve(hostname)> if 0.0.0.0
Tags []string // empty, or includes 'udp' if udp
Attrs map[string]string // any remaining service metadata from environment
}
Most of these (except `IP` and `Port`) can be overridden by container environment metadata variables prefixed with `SERVICE_` or `SERVICE_<internal-port>_`. You use a port in the key name to refer to a particular port's service. Metadata variables without a port in the name are used as the default for all services or can be used to conveniently refer to the single exposed service.
### Simple example with defaults
$ docker run -d --name redis.0 -p 10000:6379 dockerfile/redis
Will result in a service:
Results in `Service`:
{
"id": "<nodename>/redis.0:6379",
"name": "redis",
"port": 10000,
"tags": []
"ID": "hostname:redis.0:6379",
"Name": "redis",
"Port": 10000,
"IP": "192.168.1.102",
"Tags": [],
"Attrs": {}
}
The service ID is a unique identifier for this service instance. It's produced by the Consul agent's node name (often the hostname), then the name of the container, then the exposed port. You rarely need to use the ID since Consul lookups are done by name.
### Simple example with metadata
You can override service name by setting the environment variable `SERVICE_NAME`. You also don't have to specify a host port, as it will use the automatically assigned one if not provided.
$ docker run -d --name redis.0 -p 10000:6379 \
-e "SERVICE_NAME=db" \
-e "SERVICE_TAGS=master,backups" \
-e "SERVICE_REGION=us2" dockerfile/redis
$ docker run -d --name redis.0 -e "SERVICE_NAME=db" -p 6379 dockerfile/redis
Results in the service:
Results in `Service`:
{
"id": "<nodename>/redis.0:6379",
"name": "db",
"port": 23210,
"tags": []
"ID": "hostname:redis.0:6379",
"Name": "db",
"Port": 10000,
"IP": "192.168.1.102",
"Tags": ["master", "backups"],
"Attrs": {"region": "us2"}
}
You can also specify tags with a comma-delimited list. If you publish a port on UDP, it will automatically get a `udp` tag.
### Complex example with defaults
$ docker run -d --name consul -p 53/udp -e "SERVICE_TAGS=dns,backup" progrium/consul
$ docker run -d --name nginx.0 -p 4443:443 -p 8000:80 progrium/nginx
Results in the service:
{
"id": "<nodename>/consul:53",
"name": "consul",
"port": 18279,
"tags": ["dns", "backup", "udp"]
}
### Multiple published ports
If a container publishes more than one port, a service will be created for each published port. By default, the services will be named using the base name of the image and the *internal* exposed port. For example:
$ docker run -p 8000:80 -p 4443:443 --name nginx.0 progrium/nginx
Results in two services:
Results in two `Service` objects:
[
{
"id": "<nodename>/nginx.0:80",
"name": "nginx-80",
"port": 8000,
"tags": []
"ID": "hostname:nginx.0:443",
"Name": "nginx-443",
"Port": 4443,
"IP": "192.168.1.102",
"Tags": [],
"Attrs": {},
},
{
"id": "<nodename>/nginx.0:443",
"name": "nginx-443",
"port": 4443,
"tags": []
"ID": "hostname:nginx.0:80",
"Name": "nginx-80",
"Port": 8000,
"IP": "192.168.1.102",
"Tags": [],
"Attrs": {}
}
]
You can override each port's service name by setting the environment variable `SERVICE_{port}_NAME` where port is the *internal* exposed port. For example:
### Complex example with metadata
$ docker run -p 8000:80 -p 4443:443 --name nginx.0 -e "SERVICE_80_NAME=http" -e "SERVICE_443_NAME=https" progrium/nginx
$ docker run -d --name nginx.0 -p 4443:443 -p 8000:80 \
-e "SERVICE_443_NAME=https" \
-e "SERVICE_443_ID=https.12345" \
-e "SERVICE_80_NAME=http" \
-e "SERVICE_TAGS=www" progrium/nginx
Resulting in:
Results in two `Service` objects:
[
{
"id": "<nodename>/nginx.0:80",
"name": "http",
"port": 8000,
"tags": []
"ID": "https.12345",
"Name": "https",
"Port": 4443,
"IP": "192.168.1.102",
"Tags": ["www"],
"Attrs": {},
},
{
"id": "<nodename>/nginx.0:443",
"name": "https",
"port": 4443,
"tags": []
"ID": "hostname:nginx.0:80",
"Name": "http",
"Port": 8000,
"IP": "192.168.1.102",
"Tags": ["www"],
"Attrs": {}
}
]
Setting tags or any future service attributes would use the same prefix convention for multi service containers (ie `SERVICE_80_TAGS`).
## Adding support for other service registries
As you can see by either the Consul or etcd source files, writing a new registry backend is easy. Just follow the example set by those two. It boils down to writing an object that implements this interface:
type ServiceRegistry interface {
Register(service *Service) error
Deregister(service *Service) error
}
Then add your constructor (for example `NewZookeeperRegistry`) to the factory looking function in `registrator.go`.
## Todo
* Support custom Consul checks with SERVICE_CHECK_SCRIPT and SERVICE_CHECK_INTERVAL variables
* Consul backend: support custom checks with SERVICE_CHECK_SCRIPT and SERVICE_CHECK_INTERVAL variables
## Sponsors and Thanks

154
bridge.go Normal file
View File

@ -0,0 +1,154 @@
package main
import (
"log"
"net"
"os"
"path"
"strconv"
"strings"
"sync"
dockerapi "github.com/fsouza/go-dockerclient"
)
type PublishedPort struct {
HostPort string
HostIP string
ExposedPort string
PortType string
}
type Service struct {
ID string
Name string
Port int
IP string
Tags []string
Attrs map[string]string
}
func NewService(container *dockerapi.Container, port PublishedPort, isgroup bool) *Service {
defaultName := path.Base(container.Config.Image)
if isgroup {
defaultName = defaultName + "-" + port.ExposedPort
}
hostname, err := os.Hostname()
if err != nil {
hostname = port.HostIP
} else {
if port.HostIP == "0.0.0.0" {
ip, err := net.ResolveIPAddr("ip", hostname)
if err == nil {
port.HostIP = ip.String()
}
}
}
metadata := serviceMetaData(container.Config.Env, port.ExposedPort)
service := new(Service)
service.ID = hostname + ":" + contianer.Name[1:] + ":" + port.ExposedPort
service.Name = mapdefault(metadata, "name", defaultName)
p, _ := strconv.Atoi(port.HostPort)
service.Port = p
service.IP = port.HostIP
service.Tags = make([]string, 0)
tags := mapdefault(metadata, "tags", "")
if tags != "" {
service.Tags = append(service.Tags, strings.Split(tags, ",")...)
}
if port.PortType == "udp" {
service.Tags = append(service.Tags, "udp")
service.ID = service.ID + ":udp"
}
id := mapdefault(metadata, "id", "")
if id != "" {
service.ID = id
}
delete(metadata, "id")
delete(metadata, "tags")
delete(metadata, "name")
service.Attrs = metadata
return service
}
func serviceMetaData(env []string, port string) map[string]string {
metadata := make(map[string]string)
for _, kv := range env {
kvp := strings.SplitN(kv, "=", 2)
if strings.HasPrefix(kvp[0], "SERVICE_") && len(kvp) > 1 {
key := strings.ToLower(strings.TrimPrefix(kvp[0], "SERVICE_"))
portkey := strings.SplitN(key, "_", 2)
_, err := strconv.Atoi(portkey[0])
if err == nil && len(portkey) > 1 {
if portkey[0] != port {
continue
}
metadata[portkey[1]] = kvp[1]
} else {
metadata[key] = kvp[1]
}
}
}
return metadata
}
type RegistryBridge struct {
sync.Mutex
docker *dockerapi.Client
registry ServiceRegistry
services map[string][]*Service
}
func (b *RegistryBridge) Add(containerId string) {
b.Lock()
defer b.Unlock()
container, err := b.docker.InspectContainer(containerId)
if err != nil {
log.Println("registrator: unable to inspect container:", containerId, err)
return
}
ports := make([]PublishedPort, 0)
for port, published := range container.NetworkSettings.Ports {
if len(published) > 0 {
p := strings.Split(string(port), "/")
ports = append(ports, PublishedPort{published[0].HostPort, p[0], p[1]})
}
}
for _, port := range ports {
service := NewService(container, port, len(ports) > 1)
err := retry(func() error {
return b.registry.Register(service)
})
if err != nil {
log.Println("registrator: unable to register service:", service, err)
continue
}
b.services[container.ID] = append(b.services[container.ID], service)
log.Println("registrator: added:", container.ID[:12], service.ID)
}
}
func (b *RegistryBridge) Remove(containerId string) {
b.Lock()
defer b.Unlock()
for _, service := range b.services[containerId] {
err := retry(func() error {
return b.registry.Deregister(service)
})
if err != nil {
log.Println("registrator: unable to deregister service:", service.ID, err)
continue
}
log.Println("registrator: removed:", containerId[:12], service.ID)
}
delete(b.services, containerId)
}

67
consul.go Normal file
View File

@ -0,0 +1,67 @@
package main
import (
"log"
"net/url"
"github.com/armon/consul-api"
)
type ConsulRegistry struct {
client *consulapi.Client
path string
}
func NewConsulRegistry(uri *url.URL) ServiceRegistry {
config := consulapi.DefaultConfig()
if uri.Host != "" {
config.Address = uri.Host
}
client, err := consulapi.NewClient(config)
assert(err)
return &ConsulRegistry{client: client, path: uri.Path}
}
func (r *ConsulRegistry) Register(service *Service) error {
if r.path == "" || r.path == "/" {
return r.registerWithCatalog(service)
} else {
return r.registerWithKV(service)
}
}
func (r *ConsulRegistry) registerWithCatalog(service *Service) error {
registration := new(consulapi.AgentServiceRegistration)
registration.ID = service.ID
registration.Name = service.Name
registration.Port = service.Port
registration.Tags = service.Tags
// TODO registration.Check
return r.client.Agent().ServiceRegister(registration)
}
func (r *ConsulRegistry) registerWithKV(service *Service) error {
path := r.path + "/" + service.Name + "/" + service.ID
port, _ := strconv.Itoa(service.Port)
addr := net.JoinHostPort(service.IP, port)
_, err := r.client.KV().Put(consulapi.KVPair{Key: path, Value: []byte(addr)}, nil)
return err
}
func (r *ConsulRegistry) Deregister(service *Service) error {
if r.path == "" || r.path == "/" {
return r.deregisterWithCatalog(service)
} else {
return r.deregisterWithKV(service)
}
}
func (r *ConsulRegistry) deregisterWithCatalog(service *Service) error {
return r.client.Agent().ServiceDeregister(service.ID)
}
func (r *ConsulRegistry) deregisterWithKV(service *Service) error {
path := r.path + "/" + service.Name + "/" + service.ID
_, err := r.client.KV().Delete(path, nil)
return err
}

View File

@ -1,183 +0,0 @@
package main
import (
"flag"
"log"
"os"
"path"
"strconv"
"strings"
"sync"
"github.com/armon/consul-api"
"github.com/cenkalti/backoff"
dockerapi "github.com/fsouza/go-dockerclient"
)
func getopt(name, def string) string {
if env := os.Getenv(name); env != "" {
return env
}
return def
}
func assert(err error) {
if err != nil {
log.Fatal("docksul:", err)
}
}
func containerServiceData(container *dockerapi.Container, prefix, key, dfault string) string {
if prefix != "" {
key = "SERVICE_" + prefix + "_" + key
} else {
key = "SERVICE_" + key
}
for _, env := range container.Config.Env {
kv := strings.SplitN(env, "=", 2)
if strings.ToLower(kv[0]) == strings.ToLower(key) {
return kv[1]
}
}
return dfault
}
type Bridge struct {
sync.Mutex
docker *dockerapi.Client
consul *consulapi.Client
nodeName string
services map[string][]*consulapi.AgentServiceRegistration
}
func (b *Bridge) buildService(container *dockerapi.Container, hostPort, exposedPort, portType string, multiService bool) *consulapi.AgentServiceRegistration {
var keyPrefix, defaultName string
defaultName = path.Base(container.Config.Image)
if multiService {
keyPrefix = exposedPort
defaultName = defaultName + "-" + exposedPort
}
service := new(consulapi.AgentServiceRegistration)
service.ID = b.nodeName + "/" + container.Name[1:] + ":" + exposedPort
service.Name = containerServiceData(container, keyPrefix, "name", defaultName)
p, _ := strconv.Atoi(hostPort)
service.Port = p
service.Tags = make([]string, 0)
if portType == "udp" {
service.ID = service.ID + "/udp"
service.Tags = append(service.Tags, "udp")
}
tags := containerServiceData(container, keyPrefix, "tags", "")
if tags != "" {
service.Tags = append(service.Tags, strings.Split(tags, ",")...)
}
return service
}
func (b *Bridge) Add(containerId string) {
b.Lock()
defer b.Unlock()
container, err := b.docker.InspectContainer(containerId)
if err != nil {
log.Println("docksul: unable to inspect container:", containerId, err)
return
}
portDefs := make([][]string, 0)
for port, published := range container.NetworkSettings.Ports {
if len(published) > 0 {
p := strings.Split(string(port), "/")
portDefs = append(portDefs, []string{published[0].HostPort, p[0], p[1]})
}
}
multiservice := len(portDefs) > 1
for _, port := range portDefs {
service := b.buildService(container, port[0], port[1], port[2], multiservice)
err := backoff.Retry(func() error {
return b.consul.Agent().ServiceRegister(service)
}, backoff.NewExponentialBackOff())
if err != nil {
log.Println("docksul: unable to register service:", service, err)
continue
}
b.services[container.ID] = append(b.services[container.ID], service)
log.Println("docksul: added:", container.ID[:12], service.ID)
}
}
func (b *Bridge) Remove(containerId string) {
b.Lock()
defer b.Unlock()
for _, service := range b.services[containerId] {
err := backoff.Retry(func() error {
return b.consul.Agent().ServiceDeregister(service.ID)
}, backoff.NewExponentialBackOff())
if err != nil {
log.Println("docksul: unable to deregister service:", service.ID, err)
continue
}
log.Println("docksul: removed:", containerId[:12], service.ID)
}
delete(b.services, containerId)
}
func main() {
flag.Parse()
consulConfig := consulapi.DefaultConfig()
if flag.Arg(0) != "" {
consulConfig.Address = flag.Arg(0)
}
consul, err := consulapi.NewClient(consulConfig)
assert(err)
docker, err := dockerapi.NewClient(getopt("DOCKER_HOST", "unix:///var/run/docker.sock"))
assert(err)
log.Println("docksul: Getting Consul nodename...")
var nodeName string
err = backoff.Retry(func() (e error) {
nodeName, e = consul.Agent().NodeName()
if e != nil {
log.Println(e)
}
return
}, backoff.NewExponentialBackOff())
assert(err)
bridge := &Bridge{
docker: docker,
consul: consul,
nodeName: nodeName,
services: make(map[string][]*consulapi.AgentServiceRegistration),
}
containers, err := docker.ListContainers(dockerapi.ListContainersOptions{})
assert(err)
for _, listing := range containers {
bridge.Add(listing.ID[:12])
}
events := make(chan *dockerapi.APIEvents)
assert(docker.AddEventListener(events))
log.Println("docksul: Listening for Docker events...")
for msg := range events {
switch msg.Status {
case "start":
go bridge.Add(msg.ID)
case "die":
go bridge.Remove(msg.ID)
}
}
log.Fatal("docksul: docker event loop closed") // todo: reconnect?
}

37
etcd.go Normal file
View File

@ -0,0 +1,37 @@
package main
import (
"log"
"net"
"net/url"
"strconv"
"github.com/coreos/go-etcd/etcd"
)
type EtcdRegistry struct {
client *etcd.Client
path string
}
func NewEtcdRegistry(uri *url.URL) ServiceRegistry {
urls := make([]string, 0)
if uri.Host != "" {
urls = append(urls, "http://"+uri.Host)
}
return &EtcdRegistry{client: etcd.NewClient(urls), path: uri.Path}
}
func (r *EtcdRegistry) Register(service *Service) error {
path := r.path + "/" + serviceName + "/" + serviceID
port, _ := strconv.Itoa(service.Port)
addr := net.JoinHostPort(service.IP, port)
_, err := s.client.Create(path, addr, 0)
return err
}
func (r *EtcdRegistry) Deregister(service *Service) error {
path := r.path + "/" + service.Name + "/" + service.ID
_, err := s.client.Delete(path, false)
return err
}

87
registrator.go Normal file
View File

@ -0,0 +1,87 @@
package main
import (
"flag"
"log"
"os"
"strings"
"github.com/cenkalti/backoff"
dockerapi "github.com/fsouza/go-dockerclient"
)
func getopt(name, def string) string {
if env := os.Getenv(name); env != "" {
return env
}
return def
}
func assert(err error) {
if err != nil {
log.Fatal("registrator: ", err)
}
}
func retry(fn func() error) error {
return backoff.Retry(fn, backoff.NewExponentialBackOff())
}
func mapdefault(m map[string]string, key, default_ string) string {
v, ok := m[key]
if !ok {
return default_
}
return v
}
type ServiceRegistry interface {
Register(service *Service) error
Deregister(service *Service) error
}
func NewServiceRegistry(uri *url.URL) ServiceRegistry {
factory := map[string]func(*url.URL) ServiceRegistry{
"consul": NewConsulRegistry,
"etcd": NewEtcdRegistry,
}[uri.Scheme]
if factory == nil {
log.Fatal("unrecognized registry backend: ", uri.Scheme)
}
return factory(uri)
}
func main() {
flag.Parse()
docker, err := dockerapi.NewClient(getopt("DOCKER_HOST", "unix:///var/run/docker.sock"))
assert(err)
registry := NewServiceRegistry(flag.Arg(0))
bridge := &RegistryBridge{
docker: docker,
registry: registry,
services: make(map[string][]*Service),
}
containers, err := docker.ListContainers(dockerapi.ListContainersOptions{})
assert(err)
for _, listing := range containers {
bridge.Add(listing.ID[:12])
}
events := make(chan *dockerapi.APIEvents)
assert(docker.AddEventListener(events))
log.Println("registrator: Listening for Docker events...")
for msg := range events {
switch msg.Status {
case "start":
go bridge.Add(msg.ID)
case "die":
go bridge.Remove(msg.ID)
}
}
log.Fatal("registrator: docker event loop closed") // todo: reconnect?
}

Binary file not shown.