How to disable Cloud-Init in a EL-like Cloud Image

So this one is pretty simple. However, I found a lot of misinformation along the way, so I figured that I would jot the proper (and most simple) process here.

Symptoms: an RHEL (or variant) VM that takes a very long time to boot. On the VM console, you can see the following output while the VM boot process is stalled and waiting for a timeout. Note that the message below has nothing to do with cloud-init, but it's the output that I have most often seen on the console while waiting for a VM to boot.

[106.325574} random: crng init done

Note that I have run into this issue in both OpenStack (when booting from external provider networks) and in KVM.

Upon initial boot of the VM, run the command below.

13:18:01 alvaro@lykan /home/alvaro/Documents/2post
$ sudo dnf install libguestfs libguestfs-tools openssl
Last metadata expiration check: 1:53:31 ago on Mon 16 Jul 2018 01:51:05 PM CDT.
Package libguestfs-1:1.38.2-1.fc27.x86_64 is already installed, skipping.
Package libguestfs-tools-1:1.38.2-1.fc27.noarch is already installed, skipping.
Package openssl-1:1.1.0h-3.fc27.x86_64 is already installed, skipping.
Dependencies resolved.
Nothing to do.
Complete!

13:18:26 alvaro@lykan /home/alvaro/Documents/2post
$ guestfish --rw -a ../../Downloads/CentOS-7-x86_64-GenericCloud-1805.qcow2
Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: ‘help’ for help on commands
‘man’ to read the manual
‘quit’ to quit the shell

> run
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
> list-filesystems
/dev/sda1: xfs
> mount /dev/sda1 /
> touch /etc/cloud/cloud-init.disabled
> quit

Seriously, that’s it. No need to disable or remove cloud-init services.

Cloning a Ceph client auth key

 I don't recall any reason to do this other than using the same user and auth key to authenticate in different Ceph clusters, like in a multi-backend solution, or just because things get messy when you are not using a default configuration.

Sometimes, things get easy when we use the same user and auth key on both clusters for services to connect to, so let's see some background commands for managing users, keys, and permissions:

Create a new user and auth token (cinder client example):

root@ceph-admin:~# ceph auth get-or-create client.jerry
client.jerry
key: AQAZT05WoQuzJxAAX5BKxCbPf93CwihuHo27VQ==

So as you see the key is not a parameter, in a different server this will produce a completely different key.
Just to check, print the complete list of keys:

root@ceph-admin:~# ceph auth list
installed auth entries:

osd.0
key: AQCvCbtToC6MDhAATtuT70Sl+DymPCfDSsyV4w==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQC4CbtTCFJBChAAVq5spj0ff4eHZICxIOVZeA==
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQBHCbtT6APDHhAA5W00cBchwkQjh3dkKsyPjw==
caps: [mds] allow
caps: [mon] allow *
caps: [osd] allow *
client.jerry
key: AQAZT05WoQuzJxAAX5BKxCbPf93CwihuHo27VQ==

Or print a user’s authentication key to standard output, execute the command in the following format

ceph auth print-key {TYPE}.{ID}

root@ceph-admin:~# ceph auth print-key client.jerry
AQAZT05WoQuzJxAAX5BKxCbPf93CwihuHo27VQ==

To change this in order to match with others, we need to update their keys and/or their capabilities, the import command is for this, remember their keys and their capabilities will update on existing users and create new ones, use the following format:

ceph auth import -i /path/to/keyring

The keyring file needs to be in this format, if not, the command will not work and the part of the work, it will just hang.

root@ceph-admin:~# cat jerry.key
[client.jerry]
key = AQAMP01WS8i8ERAAPspjwMzUm4SL00n+WppM6A==

Now we can update the auth key for the user jerry:

root@ceph-admin:~# auth import -i ./jerry.key
imported keyring

List again.

root@ceph-admin:~# ceph auth list
installed auth entries:

osd.0
key: AQCvCbtToC6MDhAATtuT70Sl+DymPCfDSsyV4w==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQC4CbtTCFJBChAAVq5spj0ff4eHZICxIOVZeA==
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQBHCbtT6APDHhAA5W00cBchwkQjh3dkKsyPjw==
caps: [mds] allow
caps: [mon] allow *
caps: [osd] allow *
client.jerry
key: AQAMP01WS8i8ERAAPspjwMzUm4SL00n+WppM6A==

Done, I will continue posting these little helping tricks until the last post about multi-backend ceph is out.

Export instance from OpenStack with Ceph/rbd backend

Suppose that you want to migrate an instance from different infrastructures or you want to hand over an instance information to a client, so you need to recover (export) the instance volumes information.

Step 1: Get the instance UUID.

root@ceph-admin:~# nova list | grep InstanceToExport
| 2bdda36c-f0dd-4fa5-bb8b-3df346b17002 | InstanceToExport  | SHUTOFF | -          | Shutdown    | vlan8=192.168.255.53; vlan1837=10.20.37.7; vlan1829=10.20.23.53  |

UUID from the instance is returned here: 2bdda36c-f0dd-4fa5-bb8b-3df346b17002

Step 2: Get the volume UUID from the instance, using the UUID returned in step 1

root@ceph-admin:~# cinder list | grep 2bdda36c-f0dd-4fa5-bb8b-3df346b17002
| fdb279c5-24bb-45d7-a86a-a33f4c285b5a |   in-use  |       None      | 100  |     None    |   true   | 2bdda36c-f0dd-4fa5-bb8b-3df346b17002 |

UUID from the volume is returned here: fdb279c5-24bb-45d7-a86a-a33f4c285b5a

Step 3: Search from the volume on the pool in ceph, in my case this volume is stored in the cinder-volumes pool

root@ceph-admin:~# rbd --pool cinder-volumes ls | grep fdb279c5-24bb-45d7-a86a-a33f4c285b5a
volume-fdb279c5-24bb-45d7-a86a-a33f4c285b5a

By this time you have the volume name in the pool: volume-fdb279c5-24bb-45d7-a86a-a33f4c285b5a

Step 4: Export the volume.

root@ceph-admin:~# rbd export cinder-volumes/volume-fdb279c5-24bb-45d7-a86a-a33f4c285b5a ./InstanceToExport.img
Exporting image: 100% complete...done.

root@ceph-admin:~# ll -ltrh *.img
-rw-r--r-- 1 root root 100G Feb 17 17:09 InstanceToExport.img

Step 5: Compress, so you can scp or rsync faster, this step is optional but highly recommended.

root@ceph-admin:~# gzip -9 InstanceToExport.img
root@ceph-admin:~# ll *.gz
-rw-r--r-- 1 root root 1.2G Feb 17 18:02 InstanceToExport.img.gz

Step 6: Checksum, to be sure that you don't have any problem copying

root@ceph-admin:~# md5sum InstanceToExport.img >InstanceToExport.img.md5
root@ceph-admin:~# md5sum InstanceToExport.img.gz >InstanceToExport.img.gz.md5
root@ceph-admin:~# cat InstanceToExport.img.md5
5504cdf2261556135811fdd5787b33a5  InstanceToExport.img
root@ceph-admin:~# cat InstanceToExport.img.gz.md5
8a76c28d404f44cc43872e69c9965cd2  InstanceToExport.img.gz

Note: the md5sum InstanceToExport.img is going to take a lot! in my volume (100G) like 20 minutes, omit it if you want.

Why companies should embrace OSS and the DevOps movement

It’s not a secret that the best and most competitive technologies today exist in the world are based on some Open Source component, maybe the Linux kernel, GNU/Linux operating system, a version of BSD, modules, drivers, or the programming language is completely free or have a free compiler or interpreter.

On the other hand, we have a complex and extensive range of solutions that are born almost with every blink, we need options to integrate these solutions into existing technologies, we have to interconnect new software with hardware and almost all possible combinations can generate with these, so basically, no matter what kind of hardware or software want or have to work, if we want to survive in the era of cloud solutions, build an interface to interconnect them will always be the fastest solution, we will always have to be interconnected and this is a main principle that in cloud architectures is required to satisfy, where the hardware is defined by software and everything is “as a Service” (XaaS), everything has to be able to be interconnected with something else, in short, this is the Application Programming Interfaces (API) age.

Nowadays technologies are needed with API (REST, SOAP), communities (Reddit, IRC, …) accessible information (Blogs, Wikis), otherwise we have to be able to build there, the faster way possible, we need tools, and languages, plugins, everything that we can use to build these interconnections and better solutions, the only platform we can use to accomplish this with the speed needed is the Open Source, it’s not a mystery that Open Source technologies based move much faster than any other kind of proprietary technologies, so if we don’t want to be a technological dinosaur from one day to another we have to know about agile development languages (python, ruby, groovy, …), collaborative work applications (gitlab, github, trac, bugzilla, …), source code management and revision control (git, svn, …), tools that move and help us with the speed required to build new products, today knowing about Open Source, licenses, programming languages, communities is no longer an option.

Speed is not the only thing that Open Source gives, for any professional, having software freedom without limits, whether that solves the problem 100% or having a piece of software that delivers a solid foundation in order to modify and make what is required, POCs without asking for a copy of the software to a company is priceless, which also has an impact on the number of users downloading the same software, which can modify, test, add new characteristics.

I don’t want to expose a vision where nothing else exists besides the Open Source Software but to compete technologically we have to know the ecosystem or even better to innovate must know the tools and work with the right people for the job, people who can integrate all kinds of solutions, but who are this guys? These guys are like super-sysadmins + developers + Open Source gurus, all this and more, equals DevOps engineers (like me), better check this post by the puppetlabs people maybe in the future I’ll write my own.

But you don’t need to believe me, I can challenge you to find a job offer in a company that wants to innovate (any real IT company), regardless of the language or the country you are not looking for DevOps guys or Open Source knowledge.

Let’s cut to the chase, any company that wants to innovate technologically needs DevOps in its payroll, and any DevOps who wants to have a decent job requires Open Source knowledge.

Hope you enjoyed the reading, see you soon!!!

$ commit