Kubernetes networking under the hood (I)

Why is networking one of the biggest headaches for almost any opensource project? Whenever you look, OpenStack, Docker, Kubernetes… the number one issue has always been the network layer. I want to believe that the reason for that is because networking has been one of the last stack layers to be fully virtualized. And, until very recently, it was, literally, a bunch of wires (physical ones) connected to a black box (called switch or hub or router) in the top of the rack. A simple tcpdump was more than enough to debug network issues in most of the cases. We can say goodbye to those good old days, cause they are gone… but things are easier than you think.

Kubernetes networking model

If there was only one thing that you would learn from this post it should be that every pod has its own IP address and can communicate with any other pods or host in the cluster.  With plain containers deployments,  you need to link the different containers and keep track of all the mappings between the host ports and the container services: a total nightmare once you start to scale. Using other orchestration technologies like Docker Swarm or Mesos, will make this step really simple.

Our Deployment

So, with all this in mind, how this really works internally, at networking layer?  I’m not going to enter into details of how to deploy a kubernetes cluster, but I’ll be using my Raspberry Py cluster (k8s RPI cluster) with HypriotOS (a Debian/Raspbian derivate distribution) + kubeadm + flannel.

rpicluster

The only “tricky” step in the deployment is that we need to use a slightly modified deployment file for flannel (use the official arm Docker images instead of amd64 and use hostgw as the default flannel plugin instead of vlanx – we’ll be playing with vxlan and other backends and network plugins in future posts)

curl -sSL https://rawgit.com/coreos/flannel/v0.9.0/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | sed "s/vxlan/host-gw/"

In total, we’ll have 3 hosts: 1 master and 2 nodes.

net-screen1

Every host will get its own address space to assign to its pods. Flannel will be the one responsible for keeping track of which host has which net address spaces, by storing that data in the etcd service or the kubernetes API.

$ kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, podCIDR: .spec.podCIDR}' 
{
 "name": "rpi-master",
 "podCIDR": "10.244.0.0/24"
}
{
 "name": "rpi-node-1",
 "podCIDR": "10.244.1.0/24"
}
{
 "name": "rpi-node-2",
 "podCIDR": "10.244.2.0/24"
}

Finally, to add some load and pods to our cluster, an httpd service will be deployed:

kubectl run httpd --image=httpd --replicas=3 --port=80

Basic Topology

With 3 replicas, the pods will be distributed among the two nodes, and each of them will have a unique IP address.

net2

net1

With the host-gw flannel backend, the network configuration is straightforward. Everything is transparent, all is done via ip routes, and all the routes will be managed by the flannel daemon.

​root@rpi-node-1:/home/pirate# ip route
default via 192.168.0.1 dev eth0 
10.244.0.0/24 via 192.168.0.101 dev eth0 
10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1 
10.244.2.0/24 via 192.168.0.112 dev eth0 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.111

For this to work, all nodes should be on the same network. For more complex clusters, other backends like vxlan should be used to avoid those limitations, but at the expense of performance and complexity. There is no overlay protocol, natting, whatever… just a bridge connecting two different networks.

How is this managed in code

func (n *network) Run(ctx context.Context) {
	wg := sync.WaitGroup{}

	log.Info("Watching for new subnet leases")
	evts := make(chan []subnet.Event)
	wg.Add(1)
	go func() {
		subnet.WatchLeases(ctx, n.sm, n.lease, evts)
		wg.Done()
	}()

	// Store a list of routes, initialized to capacity of 10.
	n.rl = make([]netlink.Route, 0, 10)
	wg.Add(1)

	// Start a goroutine which periodically checks that the right routes are created
	go func() {
		n.routeCheck(ctx)
		wg.Done()
	}()

	defer wg.Wait()

	for {
		select {
		case evtBatch := <-evts:
			n.handleSubnetEvents(evtBatch)

		case <-ctx.Done():
			return
		}
	}
}

Like everything with this backend, all is pretty forward in flannel. In the main execution thread, we have two goroutines, one that will watch for network leases and changes (subnet.WatchLeases(), lines 57-60), and another one that will make sure that we have the desired routes in our system. (n.routeCheck(), lines 67-70). Then, the execution enters in a loop (lines 75-81) where we wait for an event to occur (notified by the previously mentioned subnet.WatchLeases() goroutine, through the evts channel) or for a ctx.Done() indicating that the program needs to terminate.

func (n *network) routeCheck(ctx context.Context) {
	for {
		select {
		case <-ctx.Done():
			return
		case <-time.After(routeCheckRetries * time.Second):
			n.checkSubnetExistInRoutes()
		}
	}
}

func (n *network) checkSubnetExistInRoutes() {
	routeList, err := netlink.RouteList(nil, netlink.FAMILY_V4)
	if err == nil {
		for _, route := range n.rl {
			exist := false
			for _, r := range routeList {
				if r.Dst == nil {
					continue
				}
				if routeEqual(r, route) {
					exist = true
					break
				}
			}
			if !exist {
				if err := netlink.RouteAdd(&route); err != nil {
					if nerr, ok := err.(net.Error); !ok {
						log.Errorf("Error recovering route to %v: %v, %v", route.Dst, route.Gw, nerr)
					}
					continue
				} else {
					log.Infof("Route recovered %v : %v", route.Dst, route.Gw)
				}
			}
		}
	} else {
		log.Errorf("Error fetching route list. Will automatically retry: %v", err)
	}
}

The routeCheck goroutine is also quite simple. Every 10 seconds (line 181) it calls n.checkSubnetExistInRoutes() and confirms that the desired routes are present in the host. If not, it just tries to add them (line 202). All the real magic happens in n.handleSubnetEvents(evtBatch) (line 77), so I consider this check a simple safety net just in case something goes really bad. (An external program or BOFH deleting routes, maybe)

func (n *network) handleSubnetEvents(batch []subnet.Event) {
...
    case subnet.EventAdded:
    ...
        n.addToRouteList(route)
        } else if err := netlink.RouteAdd(&route); err != nil {
    ...
    case subnet.EventRemoved:
    ...
        n.removeFromRouteList(route)
        if err := netlink.RouteDel(&route); err != nil {
    ...

Full code on Github.
We have two different types of events which are self-explanatory: EventAddded and EventRemoved. And after some checking, e.g., that the route exists (or not), we just add it to the host and to the list of expected routes (used by the previously mentioned routeCheck() line 176) or removed it from both places.

FORWARD default DROP policy

With the release of Docker 1.13 (+ info), there was a change in the default behavior of the FORWARD chain policy. It used to be ACCEPT, but now it has changed to DROP.

root@rpi-node-1:/home/pirate# iptables -L FORWARD
Chain FORWARD (policy DROP)

With this, all the network packets that need to go through (like our route 10.244.0.0/24 via 192.168.0.101 dev eth0) will be drop by default if there is no other matching rule (most likely case). This “small” change has bitten a lot of upgrades and new deployments. The issue affects both host-gw and vxlan flannel backends among others. Hopefully, there are several bug reports about it and a solution it’s been testing to be available soon.

https://github.com/coreos/flannel/pull/872

And now what…

For the next posts of this series, I plan on talking about other aspects of k8s networking internal likes services, ingress, load balancers, other flannel backends like VXLAN, alternatives fo flannel and VXLAN like Geneve, and even CNI, the specification and library that allows us to change between different networking plugins easily.

Stay tuned!

Advertisement

Some extra commands to partially build kubernetes

Sometimes we are just testing small changes in a command or service and we don’t need to build the whole kubernetes universe as we did in the previous post, so the alternative is to build the binary we are playing with. For that, we just need to specify it with the environment variable WHAT, and we have two ways of doing it: compile it in our own environment with make WHAT=cmd/kubelet or building it using the docker images provided by the k8s community. For this last case, we simply need to append any command we want to run with build/run.sh

build/run.sh

KUBE_RUN_COPY_OUTPUT="${KUBE_RUN_COPY_OUTPUT:-y}"

kube::build::verify_prereqs
kube::build::build_image

if [[ ${KUBE_RUN_COPY_OUTPUT} =~ ^[yY]$ ]]; then
  kube::log::status "Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable."
else
  kube::log::status "Output from this container will NOT be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=y to enable."
fi

kube::build::run_build_command "$@"

if [[ ${KUBE_RUN_COPY_OUTPUT} =~ ^[yY]$ ]]; then
  kube::build::copy_output
fi

This script is really simple to follow and very similar to the build/release.sh we talked about in the previous post.  verify_prereqs and build_image will check that we have a running docker and will build the docker image (using build/build-image/Dockerfile) Next it will run kube::build::run_build_command with the commands we want to execute inside the docker image, and finally copy the results to the _output folder.

Simple, clean and easy to follow. We will use this build/run.sh not just for building binaries, but also for anything that we want to run inside the docker environment like testing and verifying. Again, using the WHAT environment variable, we can reduce the scope of the command to just the part of the project we are working with.

A walk through Kubernetes build process

Building Kubernetes binaries by hand can be a difficult operation and you will probably end with small differences for every new try. Luckily for us, the Kubernetes community provides us with a set of tools to make things easier and we will be able to have reproducible builds, every run, even compared to the official ones. This is possible because the official releases are built using Docker containers which will be a requirement to build k8s (usually a local installation, but could also be a remote one).

Before going deep into the build process, worth noticing that thekube-buildDocker image is built based on build/build-image/Dockerfile and we will have 3 different containers during the build process using that image: the “build” and “rsync” containers are used for the build action and to transfer data between the container and the host; both will be deleted after every run. The “data” container will store everything necessary to support incremental builds so it will not be destroyed after each use. All this images and data will be stored in the build/directory, which will be also used for testing purposes (out of the scope of this post).

There are some works on the way to use Bazel (the open source release of the Blaze project that Google use internally), but for now, we will be using the all-mighty make. And as always, everything starts with the Makefile.

Makefile

A simple make help will show us all the options available, from building targets to testing, using bazel, verifying… (full output at make help). We will be focusing on make quick-release, but as you will see almost everything is related and we will be, at the end, decomposing the make all command.

define RELEASE_SKIP_TESTS_HELP_INFO
# Build a release, but skip tests
#
# Args:
#   KUBE_RELEASE_RUN_TESTS: Whether to run tests. Set to 'y' to run tests anyways.
#   KUBE_FASTBUILD: Whether to cross-compile for other architectures. Set to 'true' to do so.
#
# Example:
#   make release-skip-tests
#   make quick-release
endef
.PHONY: release-skip-tests quick-release
ifeq ($(PRINT_HELP),y)
release-skip-tests quick-release:
    @echo "$$RELEASE_SKIP_TESTS_HELP_INFO"
else
release-skip-tests quick-release: KUBE_RELEASE_RUN_TESTS = n
release-skip-tests quick-release: KUBE_FASTBUILD = true
release-skip-tests quick-release:
    build/release.sh
endif

As we can see from the code snippet above, the quick-release target (aka release-skip-tests) is the same as the release one, with two changes: We will not run any tests on it (KUBE_RELEASE_RUN_TESTS = n), and we will not compile the binaries for other architectures (KUBE_FASTBUILD = true); only linux/amd64 binaries will be created.

build/release.sh

KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
source "${KUBE_ROOT}/build/common.sh"
source "${KUBE_ROOT}/build/lib/release.sh"

KUBE_RELEASE_RUN_TESTS=${KUBE_RELEASE_RUN_TESTS-y}

kube::build::verify_prereqs
kube::build::build_image
kube::build::run_build_command make cross

if [[ $KUBE_RELEASE_RUN_TESTS =~ ^[yY]$ ]]; then
  kube::build::run_build_command make test
  kube::build::run_build_command make test-integration
fi

kube::build::copy_output

kube::release::package_tarballs

All the kube::build functions are located in the file ${KUBE_ROOT}/build/common.sh. verify_prereqs and build_image will check that we have a running docker and will build the docker image (using build/build-image/Dockerfile) that we need for our purpose.

After running kube::build::run_build_command make cross, we will again ignore any tests, and will copy the results from the container to our hosts using the rsync container (kube::build::copy_output) and pack the binary into the _output dir (kube::release::package_tarballs).

build/common.sh

# Run a command in the kube-build image. This assumes that the image has
# already been built.
function kube::build::run_build_command() {
  kube::log::status "Running build command..."
  kube::build::run_build_command_ex "${KUBE_BUILD_CONTAINER_NAME}" -- "$@"
}

# Run a command in the kube-build image. This assumes that the image has
# already been built.
#
# Arguments are in the form of
# <container name> <extra docker args> -- <command>
function kube::build::run_build_command_ex() {
  [[ $# != 0 ]] || { echo "Invalid input - please specify a container name." >&2; return 4; }
  local container_name="${1}"
  shift

  local -a docker_run_opts=(
    "--name=${container_name}"
    "--user=$(id -u):$(id -g)"
    "--hostname=${HOSTNAME}"
    "${DOCKER_MOUNT_ARGS[@]}"
  )

  local detach=false

  [[ $# != 0 ]] || { echo "Invalid input - please specify docker arguments followed by --." >&2; return 4; }
  # Everything before "--" is an arg to docker
  until [ -z "${1-}" ] ;do
    if [[ "$1"=="--" ]];then
      shift
      break
    fi
    docker_run_opts+=("$1")
    if [[ "$1"=="-d"||"$1"=="--detach" ]] ;then
      detach=true
    fi
    shift
  done

  # Everything after "--" is the command to run
  [[ $# != 0 ]] || { echo "Invalid input - please specify a command to run." >&2; return 4; }
  local -a cmd=()
  until [ -z "${1-}" ] ;do
    cmd+=("$1")
    shift
  done
 
  docker_run_opts+=(
    --env "KUBE_FASTBUILD=${KUBE_FASTBUILD:-false}"
    --env "KUBE_BUILDER_OS=${OSTYPE:-notdetected}"
    --env "KUBE_VERBOSE=${KUBE_VERBOSE}"
    --env "GOFLAGS=${GOFLAGS:-}"
    --env "GOLDFLAGS=${GOLDFLAGS:-}"
    --env "GOGCFLAGS=${GOGCFLAGS:-}"
  )

  # If we have stdin we can run interactive. This allows things like 'shell.sh'
  # to work. However, if we run this way and don't have stdin, then it ends up
  # running in a daemon-ish mode. So if we don't have a stdin, we explicitly
  # attach stderr/stdout but don't bother asking for a tty.

  if [[ -t 0 ]];then
    docker_run_opts+=(--interactive --tty)
  elif [[ "${detach}"==false ]];then
    docker_run_opts+=(--attach=stdout --attach=stderr)
  fi

  local -ra docker_cmd=(
    "${DOCKER[@]}" run "${docker_run_opts[@]}""${KUBE_BUILD_IMAGE}")

  # Clean up container from any previous run
  kube::build::destroy_container "${container_name}"

  "${docker_cmd[@]}""${cmd[@]}"

  if [[ "${detach}"==false ]];then
    kube::build::destroy_container "${container_name}"
  fi
}

Inside kube::build::run_build_command, we will call kube::build::run_build_command_ex "${KUBE_BUILD_CONTAINER_NAME}" -- "$@" where KUBE_BUILD_CONTAINER_NAME is the name of the container we got in the previous kube::build::verify_prereqs call and “$@” will be “make cross” in this case.

Then, we will get all the extra options that will be passed to the docker image (docker_run_opts) that will form the final docker run command, docker_cmd ( "${DOCKER[@]}" run "${docker_run_opts[@]}""${KUBE_BUILD_IMAGE}")

And with that, and the cmd to run inside the container (make cross), we have the final command we will execute: "${docker_cmd[@]}""${cmd[@]}"

From now on, everything will be executed inside the kube-build container. And we are back to the main Makefile (there will be an exact copy of our kubernetes root folder inside the container), but this time, we will just execute the cross target.

make cross (inside the kube-build container)

define CROSS_HELP_INFO
# Cross-compile for all platforms
# Use the 'cross-in-a-container' target to cross build when already in
# a container vs. creating a new container to build from (build-image)
# Useful for running in GCB.
#
# Example:
#   make cross
#   make cross-in-a-container
endef
.PHONY: cross cross-in-a-container
ifeq ($(PRINT_HELP),y)
cross cross-in-a-container:
    @echo "$$CROSS_HELP_INFO"
else
cross:
    hack/make-rules/cross.sh
cross-in-a-container: KUBE_OUTPUT_SUBPATH = $(OUT_DIR)/dockerized
cross-in-a-container:
ifeq (,$(wildcard /.dockerenv))
    @echo -e "\nThe 'cross-in-a-container' target can only be used from within a docker container.\n"
else
    hack/make-rules/cross.sh
endif
endif

This is pretty straightforward, just a call to hack/make-rules/cross.sh

hack/make-rules/cross.sh

KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
source "${KUBE_ROOT}/hack/lib/init.sh"

# NOTE: Using "${array[*]}" here is correct. [@] becomes distinct words (in
# bash parlance).

make all WHAT="${KUBE_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_SERVER_PLATFORMS[*]}"

make all WHAT="${KUBE_NODE_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_NODE_PLATFORMS[*]}"

make all WHAT="${KUBE_CLIENT_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_CLIENT_PLATFORMS[*]}"

make all WHAT="${KUBE_TEST_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_PLATFORMS[*]}"

make all WHAT="${KUBE_TEST_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_SERVER_PLATFORMS[*]}"

In hack/lib/init.sh we will source the file hack/lib/golang.sh which will be responsible to set all the KUBE_*_TARGETS and KUBE_*_PLATFORMS.

...
elif [[ "${KUBE_FASTBUILD:-}" == "true" ]]; then
  readonly KUBE_SERVER_PLATFORMS=(linux/amd64)
  readonly KUBE_NODE_PLATFORMS=(linux/amd64)
...

If you remember from our first call to make quick-release, it set KUBE_FASTBUILD = true so we will just compile the linux/amd64 binaries.

The targets (KUBE_SERVER_TARGETS) will also be specified by hack/lib/golang.sh

# The set of server targets that we are only building for Linux
# If you update this list, please also update build/BUILD.
kube::golang::server_targets() {
  local targets=(
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/cloud-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/hyperkube
    vendor/k8s.io/kube-aggregator
    vendor/k8s.io/apiextensions-apiserver
    plugin/cmd/kube-scheduler
  )
  echo"${targets[@]}"
}

And with all that set we call, again, the Makefile with the almighty all target. (Remember that we are now inside the kube-build container)

make all

define ALL_HELP_INFO
# Build code.
#
# Args:
#   WHAT: Directory names to build. If any of these directories has a 'main'
#     package, the build will produce executable files under $(OUT_DIR)/go/bin.
#     If not specified, "everything" will be built.
#   GOFLAGS: Extra flags to pass to 'go' when building.
#   GOLDFLAGS: Extra linking flags passed to 'go' when building.
#   GOGCFLAGS: Additional go compile flags passed to 'go' when building.
#
# Example:
#   make
#   make all
#   make all WHAT=cmd/kubelet GOFLAGS=-v
#   make all GOGCFLAGS="-N -l"
#     Note: Use the -N -l options to disable compiler optimizations an inlining.
#           Using these build options allows you to subsequently use source
#           debugging tools like delve.
endef
.PHONY: all
ifeq ($(PRINT_HELP),y)
all:
    @echo "$$ALL_HELP_INFO"
else
all: generated_files
    hack/make-rules/build.sh $(WHAT)
endif

Again, a simple make entry with the help and a call to hack/make-rules/build.sh with the WHAT variable being the KUBE_SERVER_TARGETS provided by hack/lib/golang.sh

hack/make-rules/build.sh

KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
KUBE_VERBOSE="${KUBE_VERBOSE:-1}"
source "${KUBE_ROOT}/hack/lib/init.sh"

kube::golang::build_binaries "$@"
kube::golang::place_bins

And from now on, in kube::golang::build_binaries, is where the Go magic happens. It sets up the environment, some Go flags, builds the kube toolchain (github.com/jteeuwen/go-bindata/go-bindata and hack/cmd/teststale) and finally builds the targets.

Conclusions

As you can see, although it looks complicated at first, everything can be reduced to a simple call to make all with a bunch of environment variables to get the output we want. Making use of a common docker image with the same set of tools across the whole community will also help us to reduce any differences that can be a real nightmare if we need to debug any faulty service.

Can you handle stressful situations?

During my quest for a new position, I’ve been talking with multiple recruiters and hiring managers and answering tons of questions about my career, management situations, codings skills, Linux internals… but there was one question that really struck me and it’s still resonating in my head: Can you handle stressful situations?

Yes, of course, I can. It’s something I’ve handled several times… I said.

Wrong! To put things in context, the position was (is) Infrastructure Lead Engineering Manager and several different departments asking you for resources you didn’t have. How to prioritize,  whose department to content first… If this was just a one-time problem, meh, just deal with it. But if this is something recurrent, then it’s the company the one that has a real problem. We are not error-prone. The more times we face this kind of situations, the more opportunities we’ll have to screw things up.  This is a scalability and lack of resources problem, and it’s something we shouldn’t allow to happen, for our own health and for the company correct performance. So the next time something like this happens, simply refuse to play the game and tackle the root of the problem.

Running kubernetes on a Raspberry pi cluster

Running kubernetes on a raspberry pi can be a little tricky. There are some many moving parts that the easiest way to do it, it is using one of the multiple installation methods for that but… what’s the fun of that?

I just wanted to get my hands dirty and know all the internals and craft a kubernetes cluster all by myself. So that is why I started the project ansible-k8s on GitHub [https://github.com/GheRivero/ansible-k8s] It does not pretend to be a production-ready deployment, but just an easy starting point to have a kubernetes cluster you can play with.

At this moment, there are a couple of missing parts, like better documentation (or documentation at all), install the dashboard… but it is more than enough to follow the playbooks and understand how all the pieces are arranged together.

rpicluster

On the plus side of this adventure, I just got my first patch approved [ https://github.com/kubernetes/contrib/pull/2202 ] into the kubernetes project, and it is not going to be the last.

starter-kit:compute (V) – DefCore

OpenStack is made-up of tens of projects and each of them with hundreds of possible configurations, making each cloud different. This leads to a situation where the mission of OpenStack, to ‘create an ubiquitous Open Source cloud computing platform’, is getting more complicated. Moving tasks and projects between different clouds providers (private or public) is a nightmare due to the differences between them.

DefCore sets base requirements by defining 1) capabilities, 2) code and 3) must-pass tests for all OpenStack products. This definition uses community resources and involvement to drive interoperability by creating the minimum standards for products labeled “OpenStack.”

RefStack is a source of tools for OpenStack interoperability testing, and what will be used to test our cloud and try to get the ‘OpenStack Compute Powered’ label.

Although the startet-compute:kit tag and the DefCore project are not directly related, we can combine them and work with both so a minimal OpenStack cloud can be deployed that is fully compatible with those products labeled ‘OpenStack’.

As usual we will create an specific user for this task, and get the required code from upstream to run the tests.

root@aio:~# useradd -m refstack
root@aio:~# passwd refstack
root@aio:~# usermod -a -G sudo refstack
refstack@aio:~# git clone git://git.openstack.org/stackforge/refstack-client

The refstack-client includes a script that will help as to prepare the environment for running all the tempest tests.

refstack@aio:~# cd refstack-client
refstack@aio:~/refstack-client# ./setup_env
refstack@aio:~/refstack-client# source .venv/bin/activate

Once the environment has been prepared, we need to create a tempest configuration file, tempest.conf, with our cloud environment data.

[auth]
tempest_roles = _member_

[identity]
auth_version = v2
admin_domain_name = Default
admin_tenant_name = stack
admin_password = minions
admin_username = gru
uri_v3 = http://aio:5000/v3
uri = http://aio:5000/v2.0

And we are ready to launch the tests. By default, refstack will executed the whole test suite, but since only the keystone identity service is available, we can limit the scope of the tests to just it.

(.venv)refstack@aio:~/refstack-client# ./refstack-client test -c ~/tempest.conf -vv -- tempest.api.identity
...
Ran 161 tests in 184.264s

OK
2015-08-20 10:36:06,643 refstack_client:272 INFO Tempest test complete.
2015-08-20 10:36:06,643 refstack_client:273 INFO Subunit results located in: .tempest/.testrepository/1
2015-08-20 10:36:06,664 refstack_client:276 INFO Number of passed tests: 161
2015-08-20 10:36:06,665 refstack_client:288 INFO JSON results saved in: .tempest/.testrepository/1.json

We have conducted a total of 161 tests, and all of them were completed successfully. We can also see how these results stack up against DefCore capabilities. We just need to upload the results to the refstack.net page and check it.

(.venv)stack@aio:~/refstack-client$ REFSTACK_URL=http://refstack.net/api ./refstack-client upload .tempest/.testrepository/1.json 
Test results will be uploaded to http://refstack.net/api. Ok? (yes/y): y
Test results uploaded!
URL: http://refstack.net/#/results/cf61b903-2589-43eb-9970-7129e17c595b
This cloud passes 1.8% (2/114) of the tests in the 2015.07 required capabilities for the OpenStack Powered Compute program.
Excluding flagged tests, this cloud passes 1.8% (2/109) of the required tests.

Compliance with 2015.07: NO

Of course we are not compliant with the OpenStack Powered Compute program (our target for our deployment), since we have just tested the keystone service, but on the plus side, we have passed all the identity service tests, so we are on the right path.

identity-v2-tokens-create [1/1]
tempest.api.identity.v2.test_tokens.TokensTest.test_create_token
identity-v3-tokens-create [1/1]
tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token

If we want to test the complete suite, you can do it easily, but at this moment it is pointless

(.venv)stack@aio:~/refstack-client$ ./refstack-client test -c ~/tempest.conf -vv --test-list https://raw.githubusercontent.com/openstack/defcore/master/2015.05/2015.05.required.txt

starter-kit:compute (IV) – Managing Keystone

Now that we have a functional and running keystone is time to start populating its database with all the users, projects and tenants that we’ll be using it the day to day. First of all, another user and his virtual environment will be created to install the ‘openstackclient’. This ‘new’ CLI is an effort to have an only command line tool to interact with all the openstack services.

root@aio:~# useradd -m stack
stack@aio:~$ virtualenv venv
stack@aio:~$ source venv/bin/activate
stack@aio:~$ pip install python-openstackclient

To bootstrap keystone, there is a special TOKEN that can be used to create the initial users. We also need to specify the URL where to find the keystone service. All this data can be stored in a file, ‘tokenrc’, and sourced when needed.

export OS_TOKEN=ADMIN
export OS_URL=http://aio:35357/v2.0

The value of the initial token is located in the keystone.conf file under the [DEFAULT] group, with an initial value of ADMIN.

[DEFAULT]
 …
 #admin_token = ADMIN

Creating users, projects and roles

A project (formerly known as tenant), can have several users, and each of this users has a role within that project. It can be ‘admin’, a simple member (by default, indicated by the role ‘_member_’), or any other role we want to create. What a role can and can’t do, is specified in the ‘policy.json’ configuration file. In this example, we will create a project called ‘stack’, an admin role, and two users, one of them being the admin of the project.

(venv)stack@aio:~$ openstack project create stack
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | None |
| enabled | True |
| id | ebae83a27e7a4409aea4311543b7dadb |
| name | stack |
+-------------+----------------------------------+

(venv)stack@aio:~$ openstack role create admin
+-------+----------------------------------+
| Field | Value |
+-------+----------------------------------+
| id | 3a933d2b3957477b9aabf1e0f32cb388 |
| name | admin |
+-------+----------------------------------+

(venv)stack@aio:~$ openstack user create gru --project stack --password minions
+------------+----------------------------------+
| Field | Value |
+------------+----------------------------------+
| email | None |
| enabled | True |
| id | 9be9ad60d4594c288f5b3305e84e3e63 |
| name | gru |
| project_id | ebae83a27e7a4409aea4311543b7dadb |
| username | gru |
+------------+----------------------------------+

(venv)stack@aio:~$ openstack role add --user gru --project stack admin
+-------+----------------------------------+
| Field | Value |
+-------+----------------------------------+
| id | 3a933d2b3957477b9aabf1e0f32cb388 |
| name | admin |
+-------+----------------------------------+

(venv)stack@aio:~$ openstack user create stuart --project stack --password banana
+------------+----------------------------------+
| Field | Value |
+------------+----------------------------------+
| email | None |
| enabled | True |
| id | 85a51778c59f4c66aed800f74cf3cf77 |
| name | stuart |
| project_id | ebae83a27e7a4409aea4311543b7dadb |
| username | stuart |
+------------+----------------------------------+

Although we will not need it now, it’s common to create a special project, usually called ‘service’, with one user per service deployed, like nova or glance, so each of then can authenticate and validate tokens using their own accounts.

(venv)stack@aio:~$ openstack project create service --description "Service Project"
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | Service Project |
| enabled | True |
| id | 7bd37e546f4741c6bfba687f35711ae5 |
| name | service |
+-------------+----------------------------------+

Finally, we need to create a service endpoint. This will be published in the Keystone service catalog, and will help services to locate other services in the deployed cloud.

(venv)stack@aio:~$ openstack service create --name keystone \
> --description "Keystone Identity Service" \
> identity
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | Keystone Identity Service        |
| enabled     | True                             |
| id          | 7796b8d0e5ff4267abf3b7766c670fb9 |
| name        | keystone                         |
| type        | identity                         |
+-------------+----------------------------------+
(venv)stack@aio:~$ openstack endpoint create --region RegionOne --publicurl "http://aio:5000/v2.0" --adminurl "http://aio:35357/v2.0" --internalurl "http://aio:5000/v2.0" keystone

+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| adminurl     | http://:$(admin_port)s/v2.0      |
| id           | 58081185adf94a82994d019043e3ab7e |
| internalurl  | http://:$(public_port)s/v2.0     |
| publicurl    | http://:$(public_port)s/v2.0     |
| region       | RegionOne                        |
| service_id   | 7796b8d0e5ff4267abf3b7766c670fb9 |
| service_name | keystone                         |
| service_type | identity                         |
+--------------+----------------------------------+

V2.0 vs V3 API ports

One of the differences between v2.0 and v3 API, is that in V3 there is no longer any distinction between the admin port (35357) and the public port (5000), so you can use either to manage the keystone service.

If you prefer to use V3 API through the openstackclient CLI, you need to specify the API version and the correct url of the service:

export OS_IDENTITY_API_VERSION=3
export OS_URL=http://aio:35357/v3

starter-kit:compute (III) – Keystone

From Keystone README.rst: “Keystone provides authentication, authorization and service discovery mechanisms via HTTP primarily for use by projects in the OpenStack family. It is most commonly deployed as an HTTP interface to existing identity systems, such as LDAP.” Basically, is the service where we will store and manage all our users and projects.

The installation is pretty straightforward.  First we’ll need to a couple of development libraries that will be needed later during the installation of the keystone requirements.

root@aio:~# apt-get install libffi-dev libssl-dev

The next step is to create a ‘keystone’ user and the default location where the configuration files will be located, ‘/etc/keystone’, with the correct permissions.

root@aio:~# useradd -m keystone 
root@aio:~# mkdir /etc/keystone 
root@aio:~# chown keystone:keystone /etc/keystone

Once everything is ready, we need to create a mysql database and a user with the appropriate privileges to connect to connect to it.

root@aio:~# mysql -u root -p 
mysql> create database keystone; GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'aio' IDENTIFIED BY 'keystone' WITH GRANT OPTION;FLUSH privileges;

Now, it’s time to clone the source from upstream, and create and activate a virtual environment where we will install keystone within all its requirements

keystone@aio:~$ git clone git://git.openstack.org/openstack/keystone 
keystone@aio:~$ virtualenv venv 
keystone@aio:~$ source venv/bin/activate 
(venv)keystone@aio:~$ cd keystone 
(venv)keystone@aio:~/keystone$ pip install pip --upgrade # The version installed by default is outdated
(venv)keystone@aio:~/keystone$ pip install -r requirements.txt 
(venv)keystone@aio:~/keystone$ pip install mysql-python 
(venv)keystone@aio:~/keystone$ python setup.py install

By default, keystone provides sample configuration files that we will need to copy to their default location:

(venv)keystone@aio:~/keystone$ cp -fr etc/* /etc/keystone/ 
(venv)keystone@aio:~/keystone$ mv /etc/keystone/keystone.conf.sample /etc/keystone/keystone.conf

The only initial change that will be needed to have a working keystone service in our environment will be to define the proper connection database string to our mysql server. Under the [database] group, we should define the connection string as:

connection = mysql://keystone:keystone@aio/keystone

After that, we need to populate the database with the required tables:

(venv)keystone@aio:~/keystone$ keystone-manage db_sync

And now we are ready to start the service:

(venv)keystone@aio:~/keystone$ keystone-all

Of course, this is not the best way to start a service, using the command line, but for our initial deployment is more than enough. We’ll see how to create our init scripts in the following posts.