Ceph recovery backfilling affecting production instances

alvaro.soto's picture

In any kind of distributed system you will have to choose between consistency, availability and partitioning, the CAP theorem states that in the presence of a network partition, one has to choose between consistency and availability, by default (default configurations) CEPH provides consistency and partitioning, just take in count that CEPH has many config options: ~860 in hammer, ~1100 in jewel, check this out, is jewel github co

Get total provisioned size from cinder volumes

alvaro.soto's picture

Quick way to get the total amount of provisioned space from cinder

alvaro@skyline.local: ~
$ cinder list --all-tenants
mysql like output :)

So to parse the output and add all the values in the Size col, use the next piped commands.

alvaro@skyline.local: ~
$ . admin-openrc.sh

alvaro@skyline.local: ~
$ cinder list --all-tenants | awk -F'|' '{print $6}' | sed 's/ //g' | grep -v -e '^$' | awk '{s+=$1} END {printf "%.0f", s}'

Final result in GB.

Keeping up to date git forked repos

alvaro.soto's picture

Quick guide to remember how to keep up to date forked repos:

First: Manage set of tracked repositories.

alvaro@skyline.local: ~/docker-openstack-cli
$ git remote -v
origin	https://github.com/alsotoes/docker-openstack-cli.git (fetch)
origin	https://github.com/alsotoes/docker-openstack-cli.git (push)

Second: Add the remote repo to work with.


Solve Ceph Clock Skew error

alvaro.soto's picture

Monitors can be severely affected by significant clock skews across the monitor nodes. This usually translates into weird behavior with no obvious cause. To avoid such issues, you should run a clock synchronization tool on your monitor nodes by default the monitors will allow clocks to drift up to 0.05 seconds.

This error can be seen using:


Cloning a Ceph client auth key

alvaro.soto's picture

I don't recall any reason to do this other than using the same user and auth key to authenticate in different Ceph clusters, like in a multi-backend solution, or just because things get messy when you are not using a default configuration.

Sometimes, things gets easy when we use the same user and auth key on both clusters for services to connect to, so lets see some background commands for managing users, keys and permissions:

Create new user and auth token (cinder client example):

Export instance from OpenStack with Ceph/rbd backend.

alvaro.soto's picture

Suppose that you want to migrate an instance from differents infrastructures or you want to handover and instance information to a client, so you need to recover (export) the instance volumes information.

Step 1: Get the instance UUID.

Testing juju environment inside LXC container

alvaro.soto's picture

I thing we pass the part of what juju is and how it works, so I'll post direct commands and configurations of how getting the environment working inside a LXC container, created just for juju, not the local configuration that creates and LXC container, in other words, out host server does not have any juju package.

Some links to read in case you need more info, or you can post a question.

Cinder Volume Transfer

alvaro.soto's picture

Let's assume you want to change ownership of volume from Tenant_A to Tenant_B.

Step 1: Tenant A will initiate an Ownership Transfer which will enable another tenant to take ownership of it.

$ source openrc Tenant_A Tenant_A
$ cinder transfer-create [volume_id]

An Authentication Key and a Transfer ID are returned here.

Step 2: Tenant B needs to accept the Transfer using the Transfer ID and The Authentication Keygenerated above.

The real problem behind highly transactional applications.

alvaro.soto's picture

An architecture trying to respond a least 10000 concurrent connections per second, is trying to solve the C10K problem, even if this is so last decade is still breaking servers, architectures and configurations, giving to sysadmins real headaches and not always because of real connections, also for basic DDoS attacks (pretty much is the same concept: lots and lots of new connections to the same service).

Why companies should embrace OSS and the DevOps movement.

alvaro.soto's picture

It’s not a secret that the best and most competitive technologies today exist in the world are based on some Open Source component, maybe the Linux kernel, GNU/Linux operating system, a version of BSD, modules, drivers, or the programming language is completely free or have a free compiler or interpreter.


Subscribe to Headup RSS
© 2010 Alvaro Soto, Send me an email.
Drupal theme by Kiwi Themes.