tag:blogger.com,1999:blog-912515857431474912024-03-08T18:30:07.953-06:00HeadupJust keep pushing forward...alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comBlogger46125tag:blogger.com,1999:blog-91251585743147491.post-5858972600849190612023-04-18T00:40:00.010-06:002023-04-21T23:58:51.274-06:00How to change download policy of repositories in Red Hat Satellite 6.3?<p><span style="font-family: georgia;">Tested on Red Hat Satellite 6.3</span></p><p><span style="font-family: georgia;"><br /></span></p><p><span style="font-family: georgia;">Issue</span></p><p><span style="font-family: georgia;">How to change the download policy of all enabled repositories in Satellite 6.3?</span></p><p><span style="font-family: georgia;">How to change the repository download policy to immediate in Satellite 6.3?</span></p><p><span style="font-family: georgia;"><br /></span></p><p><span style="font-family: georgia;">Raw</span></p><p><span style="font-family: georgia;">- Changing download policy to 'immediate'.</span></p><p><span style="font-family: georgia;"></span></p><blockquote><span style="font-family: georgia;">foreman-rake katello:change_download_policy DOWNLOAD_POLICY=immediate</span></blockquote><p></p><p><span style="font-family: georgia;"><br /></span></p><p><span style="font-family: georgia;">- Changing download policy to 'on-demand'.</span></p><p><span style="font-family: georgia;"></span></p><blockquote><span style="font-family: georgia;">foreman-rake katello:change_download_policy DOWNLOAD_POLICY=on_demand</span></blockquote><p></p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-10066334250616794792022-06-12T18:29:00.003-05:002023-03-24T18:31:40.115-06:00Converting the Image Format Using qemu-img<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">You can import an image file in VHD, VMDK, QCOW2, RAW, VHDX, QCOW, VDI, QED, ZVHD, or ZVHD2 format to HUAWEI CLOUD. Image files in other formats need to be converted before being imported. The open-source tool </span><span style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; font-weight: bolder;">qemu-img</span><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"> is provided for you to convert image file formats.</span></p><h4 id="section1" style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 1.154em; margin: 10px 0px;">Key points</h4><ul style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; list-style-image: none; margin: 0.25em 0px 0.25em 1.5em;"><li><span style="font-weight: bolder;">qemu-img</span> supports the mutual conversion of image formats VHD, VMDK, QCOW2, RAW, VHDX, QCOW, VDI, and QED.</li><li>ZVHD and ZVHD2 are self-developed image file formats and cannot be identified by <span style="font-weight: bolder;">qemu-img</span>. To convert image files to any of the two formats, use the <span style="font-weight: bolder;">qemu-img-hw</span> tool. </li><li>When you run the command to convert the format of VHD image files, use VPC to replace VHD. Otherwise, qemu-img cannot identify the image format.</li></ul><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">I'm using Fedora35 and I've already installed the package</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">$ sudo dnf provides qemu-img<br />Last metadata expiration check: 0:53:05 ago on Sun 12 Jun 2022 09:49:21 PM CDT.<br />qemu-img-2:6.1.0-5.fc35.x86_64 : QEMU command line tool for manipulating disk images<br />Repo : fedora<br />Matched from:<br />Provide : qemu-img = 2:6.1.0-5.fc35</p><p style="margin: 1em 0px;">qemu-img-2:6.1.0-14.fc35.x86_64 : QEMU command line tool for manipulating disk images<br />Repo : @System<br />Matched from:<br />Provide : qemu-img = 2:6.1.0-14.fc35</p><p style="margin: 1em 0px;">qemu-img-2:6.1.0-14.fc35.x86_64 : QEMU command line tool for manipulating disk images<br />Repo : updates<br />Matched from:<br />Provide : qemu-img = 2:6.1.0-14.fc35</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Checking package version.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">$ qemu-img -V<br />qemu-img version 6.1.0 (qemu-6.1.0-14.fc35)<br />Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Converting the image.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">$ export VMDK='wiki.vmdk'<br />$ export QCOW2='wiki.qcow2'<br />$ qemu-img convert -p -f vmdk -O qcow2 ${VMDK} ${QCOW2}<br /> (100.00/100%)</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Getting the image information.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">$ qemu-img info ${QCOW2}<br />image: wiki.qcow2<br />file format: qcow2<br />virtual size: 30 GiB (32212254720 bytes)<br />disk size: 15.8 GiB<br />cluster_size: 65536<br />Format specific information:<br /> compat: 1.1<br /> compression type: zlib<br /> lazy refcounts: false<br /> refcount bits: 16<br /> corrupt: false<br /> extended l2: false</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">And now enjoy, you can continue customizing the image or directing using it on QEMU.</p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-9943635318189265572021-10-14T18:25:00.004-05:002023-11-13T15:10:24.407-06:00Resetting a Windows guest’s Administrator password with guestfish<p style="text-align: justify;"><span face=""Open Sans", sans-serif" style="background-color: white; color: #333333; font-size: 13.008px;">DISCLAIMER: This is not my post is only a copy; in case the original gets deleted or whatever, posting on my personal blog gets more accessible for me to find it. You can find the original one in this link at the end of the post. </span></p><p style="text-align: justify;"><span face=""Open Sans", sans-serif" style="background-color: white; color: #333333; font-size: 13.008px;">I recently found myself with a Windows guest for which I didn’t have the Administrator password or any way of getting it. Nevertheless, I needed to make configuration changes to it. As I had no need to recover the old password, I was looking for a way to simply replace the Administrator password with one of my choices. </span></p><p style="text-align: justify;"><span face=""Open Sans", sans-serif" style="background-color: white; color: #333333; font-size: 13.008px;">I came across this excellent post on the topic at 4sysops.com. Option 4, the Sticky Keys trick, worked for me and is exceptionally simple to do with guestfish in Fedora. Windows has a feature called Sticky Keys, part of its suite of accessibility features. As such, it’s available before login and critical to this method. In short, pressing a specific sequence of keys will invoke the Sticky Keys program. </span></p><p style="text-align: justify;"><span face=""Open Sans", sans-serif" style="background-color: white; color: #333333; font-size: 13.008px;">We will use Guestfish to temporarily replace that program with a command shell, use the command shell to change the Administrator password, log in, and then put everything back how it was. N.B. As pointed out in the above post, Windows uses your password to encrypt various bits of data, including the Windows Vault and passwords stored in IE. Changing the Administrator password using this mechanism will make that data permanently inaccessible. </span></p><p style="text-align: justify;"><span face=""Open Sans", sans-serif" style="background-color: white; color: #333333; font-size: 13.008px;">First, we assume we have local access to the disk image from our Fedora box and that libguestfs is installed. Also, note that this is an offline process, so the guest must be shut down at this point. Attempting to do this while the guest runs will result in data corruption.</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"></p><div style="text-align: justify;"><span style="font-size: 13.008px;"># guestfish -i guest.img</span></div><div style="text-align: justify;"><span style="font-size: 13.008px;">Welcome to guestfish, the libguestfs filesystem interactive shell for editing virtual machine filesystems.</span></div><div style="text-align: justify;"><span style="font-size: 13.008px;">Type: 'help' for a list of commands 'man' to read the manual 'quit' to quit the shell</span></div><div style="text-align: justify;"><span style="font-size: 13.008px;">></span><fs style="font-size: 13.008px;"> mv /Windows/System32/sethc.exe /Windows/System32/sethc.exe.bak</fs></div><fs><div style="text-align: justify;"><span style="font-size: 13.008px;">></span><fs style="font-size: 13.008px;"> cp /Windows/System32/cmd.exe /Windows/System32/sethc.exe</fs></div><fs><div style="text-align: justify;"><span style="font-size: 13.008px;">></span><fs style="font-size: 13.008px;"> exit</fs></div></fs></fs><p></p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">You may find that the capitalization of the paths is different in your guest, but Guestfish’s tab completion should help you sort this out quite quickly. Start your guest again. When the login screen appears, press the SHIFT key 5 times. Instead of Sticky Keys, a command shell will be displayed:</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://us.mirrors.headup.ws/random/images/sticky_keys_password.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="800" height="480" src="https://us.mirrors.headup.ws/random/images/sticky_keys_password.jpg" width="640" /></a></div><br /><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: center;"><br /></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">The original post for Windows 2008 <a href="https://mdbooth.wordpress.com/2010/10/18/resetting-a-windows-guests-administrator-password-with-guestfish/" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">here</a></p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-74257215549344024602021-04-22T18:21:00.000-05:002023-03-24T18:23:39.575-06:00Can't initialize iptables table filter and nat: Permission denied<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">The best solution will be to change the container image to have an updated iptables version, but in case you can't do that, follow the next steps.</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Environment</p><ul style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; list-style-image: none; margin: 0.25em 0px 0.25em 1.5em;"><li>Red Hat OpenShift Container Platform 4.6+</li></ul><h2 style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 1.385em; margin: 10px 0px;">Issue</h2><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Executing iptables command in an application container fails with the following error.</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"> </p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><code style="font-family: monospace, monospace; font-size: 1em; margin: 0.5em 0px;">[root@pod]# iptables -L iptables v1.8.4 (legacy): can't initialize iptables table `filter': Permission denied Perhaps iptables or your kernel needs to be upgraded.</code></p><p style="margin: 1em 0px;"><code style="font-family: monospace, monospace; font-size: 1em; margin: 0.5em 0px;">[root@pod]# iptables -L -t nat iptables v1.8.4 (legacy): can't initialize iptables table `nat': Permission denied Perhaps iptables or your kernel needs to be upgraded.</code></p></blockquote><h2 style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 1.385em; margin: 10px 0px;">Resolution</h2><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Add the needed capabilities and match the SELinux denied context on audit logs on pod.spec.containers[0].securityContext.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><code style="font-family: monospace, monospace; font-size: 1em; margin: 0.5em 0px;">spec: containers: securityContext: privileged: false capabilities: drop: ["all"] add: ["NET_ADMIN", "NET_RAW", "NET_BIND_SERVICE"] seLinuxOptions: user: "system_u" role: "system_r" type: "container_t" level: "s0:c981,c991"</code></p></blockquote><h2 style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 1.385em; margin: 10px 0px;">Diagnostic Steps</h2><ol style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 0.25em 0px 0.25em 2em; padding: 0px;"><li>Find the worker node from where the pod is running.</li><li>Connect to the worker node.</li><li>Tail audit log.</li><li>Initialize a bash session on the pod.</li><li>Execute iptables command.</li><li>Wait for iptables denial error on audit log.</li></ol><div><span style="color: #333333; font-family: Open Sans, sans-serif;"><span style="font-size: 13.008px;"><br /></span></span></div><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">[root@worker] # tail -f /var/logs/audit/audit.log ...[ SNIP ]... type=AVC msg=audit(1618591176.860:2303): avc: denied { module_request } for pid=912615 comm="iptables" kmod="iptable_filter" scontext=system_u:system_r:container_t:s0:c981,c991 tcontext=system_u:system_r:kernel_t:s0 tclass=system permissive=0 type=AVC msg=audit(1618591176.860:2304): avc: denied { module_request } for pid=912615 comm="iptables" kmod="iptable_filter" scontext=system_u:system_r:container_t:s0:c981,c991 tcontext=system_u:system_r:kernel_t:s0 tclass=system permissive=0 ...[ SNIP ]...</span>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-22983737255694865982020-06-29T16:44:00.000-05:002023-03-25T02:04:15.258-06:00TripleO Container steps<p style="text-align: justify;"><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 1.154em;">Container steps</span></p><div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item" style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"><div class="section" id="container-steps"><p style="margin: 1em 0px;"></p><div style="text-align: justify;"><span style="font-size: 13.008px;">Similar to bare metal, containers are brought up in a stepwise manner. The </span><span style="font-size: 13.008px;">current architecture supports bringing up baremetal services alongside </span><span style="font-size: 13.008px;">containers. Therefore, baremetal steps may be required depending on the service </span><span style="font-size: 13.008px;">and they are always executed before the corresponding container step.</span></div><p></p><p style="margin: 1em 0px;"></p><div style="text-align: justify;"><span style="font-size: 13.008px;">The list below represents the correlation between the baremetal and the </span><span style="font-size: 13.008px;">container steps. These steps are executed sequentially:</span></div><p></p><ul style="list-style-image: none; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Containers config files generated per hiera settings.</p></li><li><p style="margin: 1em 0px; text-align: justify;">Host Prep</p></li><li><p style="margin: 1em 0px; text-align: justify;">Load Balancer configuration baremetal</p><blockquote style="margin: 1em 40px;"><ul class="simple" style="list-style-image: none; list-style-type: disc; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Step 1 external steps (execute Ansible on Undercloud)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Step 1 deployment steps (Ansible)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Common Deployment steps</p><ul style="list-style-image: none; list-style-type: disc; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Step 1 baremetal (Puppet)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Step 1 containers</p></li></ul></li></ul></blockquote></li><li><p style="margin: 1em 0px; text-align: justify;">Core Services (Database/Rabbit/NTP/etc.)</p><blockquote style="margin: 1em 40px;"><ul class="simple" style="list-style-image: none; list-style-type: disc; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Step 2 external steps (execute Ansible on Undercloud)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Step 2 deployment steps (Ansible)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Common Deployment steps</p><ul style="list-style-image: none; list-style-type: disc; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Step 2 baremetal (Puppet)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Step 2 containers</p></li></ul></li></ul></blockquote></li><li><p style="margin: 1em 0px; text-align: justify;">Early Openstack Service setup (Ringbuilder, etc.)</p><blockquote style="margin: 1em 40px;"><ul class="simple" style="list-style-image: none; list-style-type: disc; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Step 3 external steps (execute Ansible on Undercloud)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Step 3 deployment steps (Ansible)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Common Deployment steps</p><ul style="list-style-image: none; list-style-type: disc; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Step 3 baremetal (Puppet)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Step 3 containers</p></li></ul></li></ul></blockquote></li><li><p style="margin: 1em 0px; text-align: justify;">General OpenStack Services</p><blockquote style="margin: 1em 40px;"><ul class="simple" style="list-style-image: none; list-style-type: disc; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Step 4 external steps (execute Ansible on Undercloud)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Step 4 deployment steps (Ansible)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Common Deployment steps</p><ul style="list-style-image: none; list-style-type: disc; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Step 4 baremetal (Puppet)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Step 4 containers (Keystone initialization occurs here)</p></li></ul></li></ul></blockquote></li><li><p style="margin: 1em 0px; text-align: justify;">Service activation (Pacemaker)</p><blockquote style="margin: 1em 40px;"><ul class="simple" style="list-style-image: none; list-style-type: disc; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Step 5 external steps (execute Ansible on Undercloud)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Step 5 deployment steps (Ansible)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Common Deployment steps</p><ul style="list-style-image: none; list-style-type: disc; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Step 5 baremetal (Puppet)</p></li><li><p style="margin: 1em 0px; text-align: justify;">Step 5 containers</p></li></ul></li></ul></blockquote></li></ul></div></div>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-91801914697910311002020-06-28T16:41:00.000-05:002023-03-25T02:03:34.370-06:00View the list of images on the undercloud docker-distribution registry<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">To view the list of images on the undercloud docker-distribution registry use the following command:</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">(undercloud) $ curl <a href="http://192.168.24.1:8787/v2/_catalog" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">http://192.168.24.1:8787/v2/_catalog</a> | jq .repositories[]</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">To view a list of tags for a specific image, use the skopeo command:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">(undercloud) $ curl -s <a href="http://192.168.24.1:8787/v2/rhosp13/openstack-keystone/tags/list" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">http://192.168.24.1:8787/v2/rhosp13/openstack-keystone/tags/list</a> | jq .tags</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">To verify a tagged image, use the skopeo command:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">(undercloud) $ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp13/openstack-keystone:13.0-44</p></blockquote>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-76065036849750690662020-06-27T17:46:00.001-05:002023-03-24T17:48:21.204-06:00Updating network configuration on the Overcloud after a deployment<p style="text-align: justify;"><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">By default, subsequent change(s) made to network configuration templates (bonding options, mtu, bond type, etc) are not applied on existing nodes when the overcloud stack is updated.</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">To push an updated network configuration add <code class="docutils literal notranslate" style="font-family: monospace, monospace; font-size: 1em; margin: 0.5em 0px;"><span class="pre">UPDATE</span></code> to the list of actions set in the <code class="docutils literal notranslate" style="font-family: monospace, monospace; font-size: 1em; margin: 0.5em 0px;"><span class="pre">NetworkDeploymentActions</span></code> parameter. (The default is <code class="docutils literal notranslate" style="font-family: monospace, monospace; font-size: 1em; margin: 0.5em 0px;"><span class="pre">['CREATE']</span></code>, to enable network configuration on stack update it must be changed to: <code class="docutils literal notranslate" style="font-family: monospace, monospace; font-size: 1em; margin: 0.5em 0px;"><span class="pre">['CREATE','UPDATE']</span></code>.)</p><ul style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; list-style-image: none; margin: 0.25em 0px 0.25em 1.5em;"><li><p style="margin: 1em 0px; text-align: justify;">Enable update of the network configuration for all roles by adding the following to <code class="docutils literal notranslate" style="font-family: monospace, monospace; font-size: 1em; margin: 0.5em 0px;"><span class="pre">parameter_defaults</span></code> in an environment file:</p><div class="highlight-default notranslate"><div class="highlight"><blockquote style="margin: 1em 40px;"><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><div style="text-align: justify;"><span class="n" style="font-size: 1em;">parameter_defaults</span><span class="p" style="font-size: 1em;">:</span></div><div style="text-align: justify;"><span style="font-size: 1em;"> </span><span class="n" style="font-size: 1em;">NetworkDeploymentActions</span><span class="p" style="font-size: 1em;">:</span><span style="font-size: 1em;"> </span><span class="p" style="font-size: 1em;">[</span><span class="s1" style="font-size: 1em;">'CREATE'</span><span class="p" style="font-size: 1em;">,</span><span class="s1" style="font-size: 1em;">'UPDATE'</span><span class="p" style="font-size: 1em;">]</span></div></pre></blockquote></div></div></li><li><p style="margin: 1em 0px; text-align: justify;">Limit the network configuration update to nodes of a specific role by using a role-specific parameter, i.e: <code class="docutils literal notranslate" style="font-family: monospace, monospace; font-size: 1em; margin: 0.5em 0px;"><span class="pre">{role.name}NetworkDeploymentActions</span></code>. For example to update the network configuration on the nodes in the Compute role, add the following to <code class="docutils literal notranslate" style="font-family: monospace, monospace; font-size: 1em; margin: 0.5em 0px;"><span class="pre">parameter_defaults</span></code> in an environment file:</p><div class="highlight-default notranslate"><div class="highlight"><blockquote style="margin: 1em 40px;"><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><div style="text-align: justify;"><span class="n" style="font-size: 1em;">parameter_defaults</span><span class="p" style="font-size: 1em;">:</span></div><div style="text-align: justify;"><span style="font-size: 1em;"> </span><span class="n" style="font-size: 1em;">ComputeNetworkDeploymentActions</span><span class="p" style="font-size: 1em;">:</span><span style="font-size: 1em;"> </span><span class="p" style="font-size: 1em;">[</span><span class="s1" style="font-size: 1em;">'CREATE'</span><span class="p" style="font-size: 1em;">,</span><span class="s1" style="font-size: 1em;">'UPDATE'</span><span class="p" style="font-size: 1em;">]</span></div></pre></blockquote></div></div></li></ul>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-32896771954603892882020-06-26T17:40:00.000-05:002023-03-25T02:05:21.120-06:00OSD refusing to start with "ERROR: osd init failed: (1) Operation not permitted"<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">The main issue is: OSD refuses to start with "ERROR: osd init failed: (1) Operation not permitted"</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Log error:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">2014-11-13 02:32:32.380964 7f977fd87780 1 journal _open /var/lib/ceph/osd/ceph-289/journal fd 21: 10736369664 bytes, block size 4096 bytes, directio = 1, aio = 1<br />2014-11-13 02:32:32.393814 7f977fd87780 1 journal _open /var/lib/ceph/osd/ceph-289/journal fd 21: 10736369664 bytes, block size 4096 bytes, directio = 1, aio = 1<br />2014-11-13 02:32:42.105930 7f977fd87780 1 journal close /var/lib/ceph/osd/ceph-289/journal<br />2014-11-13 02:32:42.112233 7f977fd87780 -1 ** ERROR: osd init failed: (1) Operation not permitted</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Resolution:</p><ul style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; list-style-image: none; margin: 0.25em 0px 0.25em 1.5em;"><li>It appears the OSD is having trouble authenticating with the monitor.</li><li>Verify the keyring file is present and correct?</li><li>By default, it is located in /var/lib/ceph/osd/ceph-<number>/keyring.</number></li><li>It should match the key returned from the command</li></ul><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"># ceph auth get osd.</p></blockquote>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-5343070538588054962019-11-21T17:38:00.001-06:002023-03-24T17:39:20.959-06:00Get IPMI IP address from OS<p> <span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">First check that you have ipmitool installed:</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">[root@lykan ~]# yum provides ipmitool Last metadata expiration check: 0:06:54 ago on Thu 21 Nov 2019 10:39:22 PM CST. ipmitool-1.8.18-10.fc29.x86_64 : Utility for IPMI control Repo : fedora Matched from: Provide : ipmitool = 1.8.18-10.fc29</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Discover:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">[root@lykan ~]# ipmitool lan print | grep "IP Address" IP Address Source : Static Address IP Address : 10.10.4.5</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">The complete information provided:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">[root@lykan ~]# ipmitool lan print Set in Progress : Set Complete Auth Type Support : NONE MD2 MD5 PASSWORD Auth Type Enable : Callback : : User : : Operator : : Admin : : OEM : IP Address Source : Static Address IP Address : 10.10.4.5 Subnet Mask : 255.255.255.0 MAC Address : xx:xx:xx:xx:xx:xx SNMP Community String : public IP Header : TTL=0x40 Flags=0x00 Precedence=0x00 TOS=0x10 BMC ARP Control : ARP Responses Disabled, Gratuitous ARP Disabled Gratituous ARP Intrvl : 2.0 seconds Default Gateway IP : 10.10.4.254 Default Gateway MAC : 00:00:00:00:00:00 Backup Gateway IP : 0.0.0.0 Backup Gateway MAC : 00:00:00:00:00:00 802.1q VLAN ID : Disabled 802.1q VLAN Priority : 0 RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,128 Cipher Suite Priv Max : XXXaaaXXaaaXaaa : X=Cipher Suite Unused : c=CALLBACK : u=USER : o=OPERATOR : a=ADMIN : O=OEM Bad Password Threshold : Not Available</p></blockquote>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-89048849501874036182019-10-01T17:35:00.001-05:002023-03-24T17:37:44.309-06:00Improve user experience using QEMU/KVM with Windows guest<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">A lot of sysadmins, SRE o wherever you want to call us, using native Linux in our laptops have the need to use virtual machines running Windows (some support, pentesting tasks, etc), if you are passionate about running periodic updates by now you figure out the main problem of this, if not, you will; the main problem is that on every kernel upgrade, you will lose the modules of VMware or VirtualBox, the best solution for this is to use QEMU/KVM, the K is for kernel so the support is embedded in the kernel, with this you will never lose support on your virtual machines, but there is a catch, even if you install virtIO drivers you will face issues like the screen does not resize, copy and paste from host to guest does not work, and is very sad to work that way.</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">So the solution: The SPICE project aims to provide a complete open-source solution for remote access to virtual machines in a seamless way so you can play videos, record audio, share USB devices, and share folders without complications.<br /><br /></p><center style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"><br /></center><center style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"><br /><img src="https://www.spice-space.org/static/images/rhel7_win7.png" style="border-style: none; height: auto; max-width: 100%; width: 883.172px;" /></center><center style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"><img src="https://www.spice-space.org/static/images/win7_rhel.png" style="border-style: none; height: auto; max-width: 100%; width: 883.172px;" /><br /></center><br style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;" /><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">SPICE could be divided into 4 different components: Protocol, Client, Server, and Guest. The protocol is the specification in the communication of the three other components; A client such as a remote viewer is responsible to send data and translate the data from the Virtual Machine (VM) so you can interact with it; The SPICE server is the library used by the hypervisor in order to share the VM under SPICE protocol; And finally, the Guest side is all the software that must be running in the VM in order to make SPICE fully functional, such as the QXL driver and SPICE VDAgent.</span><br style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;" /><br style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;" /><br style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;" /><center style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"><img src="https://www.spice-space.org/static/images/spice_schem.png" style="border-style: none; height: auto; max-width: 100%;" /></center><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br /><br /><br />Just put in your virtual machine a channel spice and install the driver, the latest version could be found <a href="https://www.spice-space.org/download/binaries/spice-guest-tools/" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">here</a>.</p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-34309413159560239962019-05-23T17:29:00.001-05:002023-03-24T17:33:30.104-06:00fake_pxe as pm_type in RHOSP13 (TripleO + OpenStack Queens)<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">So, in RHOSP13 fake_pxe is being deprecated to change in RHOSP14 for manual management, the problem is that is just in between the migration, so there is no a clean way to use fake_pxe in RHOSP13. Another change is in the installation of undercloud, the option enabled_drivers is now DEPRECATED and changed by enabled_hardware_types. What now, in order to be able to use fake_pxe as a pm_type first install the undercloud without the options enabled_drivers, only use enabled_hardware_types, and add at the end manual-management, like this:</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">... #enabled_drivers=pxe_drac,pxe_ilo,pxe_ipmitool enabled_hardware_types=redfish,ipmi,idrac,ilo,manual-management ...</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">After that just install the undercloud using the common way.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">[stack@director01 ~]$ openstack undercloud install ... ############################################################################# Undercloud install complete. The file containing this installation's passwords is at /home/stack/undercloud-passwords.conf. There is also a stackrc file at /home/stack/stackrc. These files are needed to interact with the OpenStack services, and should be secured. ############################################################################# [stack@director01 ~]$</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Next, change manually the ironic.conf file located in /etc/ironic/ironic.conf to enable the DEPRECATED option enabled_drivers and add fake as a new driver.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">enabled_drivers=pxe_drac,pxe_ilo,pxe_ipmitool,fake enabled_hardware_types=redfish,ipmi,idrac,ilo,manual-management</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">And restart ironic-conductor service:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">(undercloud) [stack@director01 ~]$ sudo systemctl restart openstack-ironic-conductor</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">check the drivers:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">(undercloud) [stack@director01 ]$ openstack baremetal driver list +---------------------+------------------------+ | Supported driver(s) | Active host(s) | +---------------------+------------------------+ | idrac | director01 | | ilo | director01 | | ipmi | director01 | | manual-management | director01 | | pxe_drac | director01 | | pxe_ilo | director01 | | pxe_ipmitool | director01 | | redfish | director01 | +---------------------+------------------------+</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Now can we add a instackenv.json file.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">(undercloud) [stack@director01 ~]$ cat instackenv-controller01.json { "nodes":[ { "mac":["controller1_mac"], "name":"nuc-controller01", "arch":"x86_64", "capabilities":"profile:control,node:controller-0,boot_option:local", "pm_type":"fake" } ] }</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">If you don't do this or try to use manual-management pm_type at this moment you will get an error similar to this one:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">(undercloud) [stack@director01 ~]$ openstack overcloud node import ~/instackenv-controller01.json Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: 6ce7871c-d9d0-448e-9b46-78ced387fa48 Waiting for messages on queue 'tripleo' with no timeout. No valid host was found. Reason: No conductor service registered which supports driver fake. (HTTP 400) Exception registering nodes: No valid host was found. Reason: No conductor service registered which supports driver fake. (HTTP 400)</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Import the new node definition to ironic and run introspection:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">(undercloud) [stack@director01 ~]$ openstack overcloud node import ~/instackenv-compute01.json Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: 434cfe01-740d-4d58-b504-6f291ab12823 Waiting for messages on queue 'tripleo' with no timeout. 1 node(s) successfully moved to the "manageable" state. Successfully registered node UUID 62ce7d2c-03ae-4c6e-8c4a-13e817f26fa3 (undercloud) [stack@director01 ~]$ (undercloud) [stack@director01 ~]$ openstack baremetal introspection start --wait nuc-controller01 Waiting for introspection to finish... +------------------+-------+ | UUID | Error | +------------------+-------+ | nuc-controller01 | None | +------------------+-------+</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">But as I said, the fake driver is not going to be supported in RHOSP14 so version 13, is in the middle of the migration and we can introspect the node using fake driver, but we are not going to be able to install it, if we tried so, we will get an error like this one:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">(undercloud) [stack@director01 ~]$ openstack action execution output show a637a01a-5f66-48a0-9e25-96700240c17e { "result": "Invalid node data: unknown pm_type (ironic driver to use): manual" }</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">So in order to solve this we need to change the driver type directly in the database, first, find the password in ironic.conf file</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">(undercloud) [stack@director01 ~]$ grep mysql /etc/ironic/ironic.conf #mysql_engine = InnoDB connection=mysql+pymysql://ironic:38315b04050cd6ad074ae75855f7c4367299b61a@192.168.10.9/ironic # set this to no value. Example: mysql_sql_mode= (string #mysql_sql_mode = TRADITIONAL #mysql_enable_ndb = false</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Then look for the drivers configured.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">MariaDB [ironic]> select name,driver from nodes; +------------------+--------+ | name | driver | +------------------+--------+ | nuc-controller01 | fake | | nuc-compute01 | fake | | nuc-compute02 | fake | +------------------+--------+ 3 rows in set (0.00 sec) MariaDB [ironic]> update nodes set driver = "manual-management" where name = "nuc-controller01"; Query OK, 1 row affected (0.01 sec) Rows matched: 1 Changed: 1 Warnings: 0 MariaDB [ironic]> update nodes set driver = "manual-management" where name = "nuc-compute01"; Query OK, 1 row affected (0.01 sec) Rows matched: 1 Changed: 1 Warnings: 0 MariaDB [ironic]> update nodes set driver = "manual-management" where name = "nuc-compute02"; Query OK, 1 row affected (0.01 sec) Rows matched: 1 Changed: 1 Warnings: 0 MariaDB [ironic]> select name,driver from nodes; +------------------+--------+ | name | driver | +------------------+--------+ | nuc-controller01 | manual-management | | nuc-compute01 | manual-management | | nuc-compute02 | manual-management | +------------------+--------+ 1 rows in set (0.00 sec)</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">After all this you can now safely continue with the common installation process, just remember when performing Overcloud deployment, check the node status with the ironic node-list command. Wait until the node status changes from deploying to deploy wait-callback and then manually power the nodes.</p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-20026695276837321392019-05-21T17:25:00.001-05:002023-03-24T17:27:30.889-06:00How to Boot into Single User Mode in CentOS/RHEL 7<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">DISCLAIMER: This is not my post is only a copy, in case the original gets deleted or whatever, posting on my personal blog gets easier for me to find it. You can find the original one at this link</span><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"> </span><a href="https://vpsie.com/knowledge-base/boot-single-user-mode-centos-rhel-vpsie/" style="color: #0074bd; font-family: "Open Sans", sans-serif; font-size: 13.008px; text-decoration-line: none;" target="_blank">https://vpsie.com/knowledge-base/boot-single-user-mode-centos-rhel-vpsie/</a></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">The first thing to do is to open Terminal and log in to your CentOS 7 server.</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">After, restarting your server wait for the GRUB boot menu to show.</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">The next step is to select your Kernel version and press</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">e</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">key to edit the first boot option. Find the kernel line (starts with “linux16“), then change the</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">ro</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">to</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">rw init=/sysroot/bin/sh .</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">When you have finished, press</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">Ctrl-X</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">or</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">F10</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">to boot into single-user mode</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">After mounting the root filesystem using the following command:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">chroot /sysroot/</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Now, to finish this process reboot your server using the following command:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">reboot -f</p></blockquote>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-39599119668072924882019-04-17T17:22:00.001-05:002023-03-24T17:24:11.588-06:00XFS online resize<p> <span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">You're working on an XFS filesystem, in this case, you need to use xfs_growfs instead of resize2fs. Two commands are needed to perform this task :</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"># growpart /dev/sda 1</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">growpart is used to expand the sda1 partition to the whole sda disk.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"># xfs_growfs -d /dev/sda1</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">xfs_growfs is used to resize and apply the changes.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"># df -h</p></blockquote>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-38571156875263889512019-04-05T17:24:00.000-06:002023-03-24T17:25:41.514-06:00Convert string <-> int64 using golang #go-nuts<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">I believe that if you are going to work with timestamps is better to do it in epoch stamps, so in GO epoch is type int64.</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;">package main
import (
"fmt"
"time"
"strconv"
)
func main() {
now := time.Now()
nanos := now.UnixNano()
bufferTimestamp := strconv.FormatInt(nanos, 10)
fmt.Printf("bufferTimestamp value: %s\n", bufferTimestamp)
timestamp, err := strconv.ParseInt(string(bufferTimestamp), 10, 64)
if err != nil {
fmt.Printf("Error: %d of type %T\n", timestamp, timestamp)
panic(err)
} else {
fmt.Printf("Converted value: %d\n", timestamp)
}
}
</pre></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">By running this you will have an output like this.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;">$ go run test/convert_stringtoint64.go
bufferTimestamp value 1556951794912716618 of type string
Converted value 1556951794912716618 of type int64</pre></blockquote>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-39317137568730474702018-12-12T17:20:00.000-06:002023-03-24T17:22:17.852-06:00How to disable Cloud-Init in a EL-like Cloud Image<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">So this one is pretty simple. However, I found a lot of misinformation along the way, so I figured that I would jot the proper (and most simple) process here.</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Symptoms: an RHEL (or variant) VM that takes a very long time to boot. On the VM console, you can see the following output while the VM boot process is stalled and waiting for a timeout. Note that the message below has nothing to do with cloud-init, but it's the output that I have most often seen on the console while waiting for a VM to boot.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">[106.325574} random: crng init done</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Note that I have run into this issue in both OpenStack (when booting from external provider networks) and in KVM.</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Upon initial boot of the VM, run the command below.</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">13:18:01 alvaro@lykan /home/alvaro/Documents/2post<br />$ sudo dnf install libguestfs libguestfs-tools openssl<br />Last metadata expiration check: 1:53:31 ago on Mon 16 Jul 2018 01:51:05 PM CDT.<br />Package libguestfs-1:1.38.2-1.fc27.x86_64 is already installed, skipping.<br />Package libguestfs-tools-1:1.38.2-1.fc27.noarch is already installed, skipping.<br />Package openssl-1:1.1.0h-3.fc27.x86_64 is already installed, skipping.<br />Dependencies resolved.<br />Nothing to do.<br />Complete!</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">13:18:26 alvaro@lykan /home/alvaro/Documents/2post<br />$ guestfish --rw -a ../../Downloads/CentOS-7-x86_64-GenericCloud-1805.qcow2<br />Welcome to guestfish, the guest filesystem shell for<br />editing virtual machine filesystems and disk images.</p><p style="margin: 1em 0px;">Type: ‘help’ for help on commands<br />‘man’ to read the manual<br />‘quit’ to quit the shell</p><p style="margin: 1em 0px;">><fs> run<br />100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00<br />><fs> list-filesystems<br />/dev/sda1: xfs<br />><fs> mount /dev/sda1 /<br />><fs> touch /etc/cloud/cloud-init.disabled<br />><fs> quit</fs></fs></fs></fs></fs></p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Seriously, that’s it. No need to disable or remove cloud-init services.</p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-64913160647229471462018-07-16T17:18:00.000-05:002023-03-24T17:20:13.305-06:00Change password to users on qcow2 disk or images<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">Sometimes you need to change the password to a user in a qcow2 image, to test locally, or if you are using an infrastructure without cloud-init, regardless of the user the procedure is the same.</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Depending on the system the packages name could change a little, I'm using Fedora 27 I have installed</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">[alvaro@lykan 2post]$ sudo dnf install libguestfs libguestfs-tools openssl<br />Last metadata expiration check: 1:53:31 ago on Mon 16 Jul 2018 01:51:05 PM CDT.<br />Package libguestfs-1:1.38.2-1.fc27.x86_64 is already installed, skipping.<br />Package libguestfs-tools-1:1.38.2-1.fc27.noarch is already installed, skipping.<br />Package openssl-1:1.1.0h-3.fc27.x86_64 is already installed, skipping.<br />Dependencies resolved.<br />Nothing to do.<br />Complete!</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br />Obviously, I have a QEMU environment to test and run the images, a very important part just to know that your steps are working.</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">[alvaro@lykan 2post]$ guestfish --rw -a ../../Downloads/CentOS-7-x86_64-GenericCloud-1805.qcow2</p><p style="margin: 1em 0px;">Welcome to guestfish, the guest filesystem shell for<br />editing virtual machine filesystems and disk images.</p><p style="margin: 1em 0px;">Type: ‘help’ for help on commands<br />‘man’ to read the manual<br />‘quit’ to quit the shell</p><p style="margin: 1em 0px;">><.fs> run<br />100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00<br />><.fs> list-filesystems<br />/dev/sda1: xfs<br />><.fs> mount /dev/sda1 /<br />><.fs> cp /etc/shadow /etc/shadow-original<br />><.fs> vi /etc/shadow</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br />Inside the vim editor, you will see the file and now you can change the hash of any user (do not close this until you reached the last step), in any other terminal run:</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">[alvaro@lykan 2post]$ openssl passwd -1 mysuperpassword<br />$1$GKdzYMMe$q20PpMv5i/QFbmgwOqtZy1</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br />Copy that generated hash and copy inside the first and second colon punctuation symbol (delete every inside this)</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br />Before</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">root:!!:17687:0:99999:7:::<br />bin:*:17632:0:99999:7:::<br />daemon:*:17632:0:99999:7:::<br />adm:*:17632:0:99999:7:::<br />lp:*:17632:0:99999:7:::<br />sync:*:17632:0:99999:7:::<br />shutdown:*:17632:0:99999:7:::<br />halt:*:17632:0:99999:7:::<br />mail:*:17632:0:99999:7:::<br />operator:*:17632:0:99999:7:::<br />games:*:17632:0:99999:7:::<br />ftp:*:17632:0:99999:7:::<br />nobody:*:17632:0:99999:7:::<br />systemd-network:!!:17687::::::<br />dbus:!!:17687::::::<br />polkitd:!!:17687::::::<br />rpc:!!:17687:0:99999:7:::<br />rpcuser:!!:17687::::::<br />nfsnobody:!!:17687::::::<br />sshd:!!:17687::::::<br />postfix:!!:17687::::::<br />chrony:!!:17687::::::</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br />After</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">root:$1$GKdzYMMe$q20PpMv5i/QFbmgwOqtZy1:17687:0:99999:7:::<br />bin:*:17632:0:99999:7:::<br />daemon:*:17632:0:99999:7:::<br />adm:*:17632:0:99999:7:::<br />lp:*:17632:0:99999:7:::<br />sync:*:17632:0:99999:7:::<br />shutdown:*:17632:0:99999:7:::<br />halt:*:17632:0:99999:7:::<br />mail:*:17632:0:99999:7:::<br />operator:*:17632:0:99999:7:::<br />games:*:17632:0:99999:7:::<br />ftp:*:17632:0:99999:7:::<br />nobody:*:17632:0:99999:7:::<br />systemd-network:!!:17687::::::<br />dbus:!!:17687::::::<br />polkitd:!!:17687::::::<br />rpc:!!:17687:0:99999:7:::<br />rpcuser:!!:17687::::::<br />nfsnobody:!!:17687::::::<br />sshd:!!:17687::::::<br />postfix:!!:17687::::::<br />chrony:!!:17687::::::</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br />Close the vim editor, save the changes, and exit guestfish</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">><.fs> quit</p><p style="margin: 1em 0px;">[alvaro@lykan 2post]$</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br />Now you can test the image on any cloud environment or using your local QEMU environment.</p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-59372207404130732262017-12-06T16:50:00.000-06:002023-03-24T16:52:50.041-06:00Get total provisioned size from cinder volumes<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">A quick way to get the total amount of provisioned space from cinder</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">alvaro@skyline.local: ~<br />$ cinder list --all-tenants<br />mysql like output :)</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">So to parse the output and add all the values in the <span style="font-weight: bolder;">Size</span> col, use the next piped commands.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">alvaro@skyline.local: ~<br />$ . admin-openrc.sh</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">alvaro@skyline.local: ~<br />$ cinder list --all-tenants | awk -F'|' '{print $6}' | sed 's/ //g' | grep -v -e '^$' | awk '{s+=$1} END {printf "%.0f", s}'<br />13453</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">The final result is in GB.</p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-29578410777562716292017-06-14T16:54:00.015-05:002023-03-24T17:02:45.418-06:00Ceph recovery backfilling affecting production instances<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">In any kind of distributed system, you will have to choose between consistency, availability, and partitioning, the</span><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"> </span><a href="https://en.wikipedia.org/wiki/CAP_theorem" style="color: #0074bd; font-family: "Open Sans", sans-serif; font-size: 13.008px; text-decoration-line: none;" target="_blank">CAP theorem</a><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"> </span><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">states that in the presence of a network partition, one has to choose between consistency and availability, by default (default configurations) CEPH provides consistency and partitioning, just take in count that CEPH has many config options: ~860 in hammer, ~1100 in jewel, check this out, is jewel github</span><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"> </span><a href="https://github.com/ceph/ceph/blob/jewel/src/common/config_opts.h" style="color: #0074bd; font-family: "Open Sans", sans-serif; font-size: 13.008px; text-decoration-line: none;" target="_blank">config_opts.h</a><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"> </span><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">file.</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">If you want any specific behavior in your cluster depends on your ability to configure and/or to change on the fly in case of contingency, this post talks about specific default recovery / backfilling option clusters, maybe you have noticed that in case of a critical failure, like losing a complete node, this causes a lot of movement of data, lots of ops on the drives, by default the cluster is going to try to recover in the fastest way possible, and also needs to support the normal operation and common use, like I said at the beginning of the post, by default CEPH have consistency and partitioning, so the common response is to start to have failures in the availability and users will start to notice high latency, high CPU usage in instances using RBD backend because of the slow response.</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><center style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"><br /><br /></center><div class="separator" style="clear: both; text-align: center;"><a href="https://us.mirrors.headup.ws/images/this_is_fine.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="168" data-original-width="300" height="168" src="https://us.mirrors.headup.ws/images/this_is_fine.jpeg" width="300" /></a></div><br /><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br /></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Try to think of this in a better way and let's analyze the problem, if we have a replica 3 cluster and we have a server down (even if we have a 3 servers cluster), the operation is still possible and the recovery jobs are no that important because CEPH will try to achieve consistency all the time, it will achieve the correct 3 replica consistency eventually, so everything will be fine, no data loss, the remaining replicas will start to regenerate the missing replica in others nodes, the big problem is the backfilling will compromise the operation, so the real problem is that we need to choose between a quick recovery or a common response to the clients and watchers connected, the response is not that hard to know, <span style="font-weight: bolder;">operation response is priority number 0</span>!!!!</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><center style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"><br /><img height="300" src="https://blog.headup.ws/zbox/images/ceph_failure_crush01.png" style="border-style: none; height: auto; max-width: 100%;" width="400" /><br /><br /><img height="300" src="https://blog.headup.ws/zbox/images/ceph_failure_crush02.png" style="border-style: none; height: auto; max-width: 100%;" width="400" /><br /><span style="font-weight: bolder;">Lost and recovery action in CRUSH (Image from Samuel Just, Vault 2015)</span><br /></center><br style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;" /><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">This is not the non-plus ultra solution, is just my solution to this problem, all this was tested in a CEPH hammer cluster:</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">1.- The better one is to configure at the beginning of the installation in the ceph.conf file<br /><br />******* SNIP *******<br />[osd]<br />....<br />osd max backfills = 1<br />osd recovery threads = 1<br />osd recovery op priority = 1<br />osd client op priority = 63<br />osd recovery max active = 1<br />osd snap trim sleep = 0.1<br />....<br />******* SNIP *******<br /></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">2.- If not, you can inject the on-the-fly options, you can use osd.x where x is the number of the osd daemon, or like the next example applies cluster-wide, but remember to put in the config file after because these options will be lost on reboot.<br /></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">ceph@stor01:~$ sudo ceph tell osd.* injectargs '--osd-max-backfills 1'<br />ceph@stor01:~$ sudo ceph tell osd.* injectargs '--osd-recovery-threads 1'<br />ceph@stor01:~$ sudo ceph tell osd.* injectargs '--osd-recovery-op-priority 1'<br />ceph@stor01:~$ sudo ceph tell osd.* injectargs '--osd-client-op-priority 63'<br />ceph@stor01:~$ sudo ceph tell osd.* injectargs '--osd-recovery-max-active 1'<br />ceph@stor01:~$ sudo ceph tell osd.* injectargs '--osd-snap-trim-sleep 0.1'</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">The final result will be a really slow recovery of the cluster, but operation without any kind of problem.</p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-85184655669416401712017-04-12T16:46:00.002-05:002023-03-24T16:50:17.094-06:00Keeping up to date git forked repos<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">A quick guide to remembering how to keep up-to-date forked repos:</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">First: Manage a set of tracked repositories.</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">alvaro@skyline.local: ~/docker-openstack-cli<br />$ git remote -v</span><br />origin <a href="https://github.com/alsotoes/docker-openstack-cli.git" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">https://github.com/alsotoes/docker-openstack-cli.git</a> (fetch)<br />origin <a href="https://github.com/alsotoes/docker-openstack-cli.git" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">https://github.com/alsotoes/docker-openstack-cli.git</a> (push)</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Second: Add the remote repo to work with.</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">alvaro@skyline.local: ~/docker-openstack-cli<br />$ git remote add kionetworks <a href="https://github.com/kionetworks/docker-openstack-cli.git" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">https://github.com/kionetworks/docker-openstack-cli.git</a></span></p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Third: Print repo local configuration.</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">alvaro@skyline.local: ~/docker-openstack-cli<br />$ git remote -v</span><br />kionetworks <a href="https://github.com/kionetworks/docker-openstack-cli.git" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">https://github.com/kionetworks/docker-openstack-cli.git</a> (fetch)<br />kionetworks <a href="https://github.com/kionetworks/docker-openstack-cli.git" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">https://github.com/kionetworks/docker-openstack-cli.git</a> (push)<br />origin <a href="https://github.com/alsotoes/docker-openstack-cli.git" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">https://github.com/alsotoes/docker-openstack-cli.git</a> (fetch)<br />origin <a href="https://github.com/alsotoes/docker-openstack-cli.git" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">https://github.com/alsotoes/docker-openstack-cli.git</a> (push)</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Fourth: Push to the remote repo, to complete the update.</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">alvaro@skyline.local: ~/docker-openstack-cli<br />$ git push kionetworks</span><br />Counting objects: 3, done.<br />Delta compression using up to 2 threads.<br />Compressing objects: 100% (3/3), done.<br />Writing objects: 100% (3/3), 311 bytes | 0 bytes/s, done.<br />Total 3 (delta 2), reused 0 (delta 0)<br />remote: Resolving deltas: 100% (2/2), completed with 2 local objects.<br />To <a href="https://github.com/kionetworks/docker-openstack-cli.git" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">https://github.com/kionetworks/docker-openstack-cli.git</a><br />51bb74b..33e5dce master -> master<br /><a href="mailto:alvaro@skyline.local" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">alvaro@skyline.local</a>: ~/docker-openstack-cli<br />hist:600 jobs:0 $</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Pull new changes from origin.</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">alvaro@skyline.local: ~/docker-openstack-cli<br />$ git pull</span><br />Already up-to-date.</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Pull new changes from a remote called <span style="font-weight: bolder;">kionetworks</span>.</span></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">alvaro@skyline.local: ~/docker-openstack-cli<br />$ git pull kionetworks master</span><br />From <a href="https://github.com/kionetworks/docker-openstack-cli" style="background-color: transparent; color: #0074bd; text-decoration-line: none;">https://github.com/kionetworks/docker-openstack-cli</a><br />* branch master -> FETCH_HEAD<br />Already up-to-date.</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Sorry if this post has too little information, is just a remember.</p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-38110418949560337142016-11-16T16:45:00.000-06:002023-03-24T16:46:54.968-06:00Solve Ceph Clock Skew error<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">Monitors can be severely affected by significant clock skews across the monitor nodes. This usually translates into weird behavior with no obvious cause. To avoid such issues, you should run a clock synchronization tool on your monitor nodes by default the monitors will allow clocks to drift up to 0.05 seconds.</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">This error can be seen using:<br /></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"># ceph -s<br /># ceph health detail</p></blockquote><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">root@ceph01:~# ceph -s<br />cluster 9227547b-bb6b-44f7-b877-3f6d25b942a4<br />health HEALTH_WARN<br /><span style="font-weight: bolder;">clock skew detected on mon.ceph01</span><br />monmap e3: 3 mons at {ceph01=172.18.3.5:6789/0,ceph02=172.18.5.6:6789/0,ceph03=172.18.5.7:6789/0}<br />election epoch 24, quorum 0,1,2 ceph01,ceph02,ceph03<br />mdsmap e17: 1/1/1 up {0=ceph01=up:active}<br />osdmap e245: 22 osds: 22 up, 22 in<br />pgmap v14727: 1408 pgs, 5 pools, 11977 MB data, 3183 objects<br />24729 MB used, 16361 GB / 16385 GB avail<br />1408 active+clean</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">The solution? just re-sync the clock in the affected mon, and restart the mon daemon.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">root@ceph01:~# </span>service ntp stop<br />* Stopping NTP server ntpd<br /><span style="font-weight: bolder;">root@ceph01:~# </span>ntpdate ntp.ubuntu.com<br />16 Nov 01:24:16 ntpdate[4149434]: adjust time server 91.189.91.157 offset -0.002235 sec<br /><span style="font-weight: bolder;">root@ceph01:~# </span>ntpd -gq<br />ntpd: time slew +0.003482s<br /><span style="font-weight: bolder;">root@ceph01:~# </span>service ntp start<br />* Starting NTP server ntpd<br /><span style="font-weight: bolder;">root@ceph01:~# </span>restart ceph-mon-all<br />ceph-mon-all start/running</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Just to be sure, sometimes it will be better if you sync the clock on all Mon</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Also, this default parameter (0.05 seconds) can be changed in the ceph config file, but that you can doesn't mean that you should, the default value is a perfect configuration.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">root@ceph01:~# </span>cat /etc/ceph/ceph.conf<br />....</p><p style="margin: 1em 0px;">[mon]<br />mon clock drift allowed = 10</p><p style="margin: 1em 0px;">...</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Check again the cluster status, sometimes it takes a few seconds, like 30 seconds.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">root@ceph01:~# </span>ceph -s<br />cluster 9227547b-bb6b-44f7-b877-3f6d25b942a4<br />health HEALTH_OK<br />monmap e3: 3 mons at {ceph01=172.18.3.5:6789/0,ceph02=172.18.5.6:6789/0,ceph03=172.18.5.7:6789/0}<br />election epoch 24, quorum 0,1,2 ceph01,ceph02,ceph03<br />mdsmap e17: 1/1/1 up {0=ceph01=up:active}<br />osdmap e245: 22 osds: 22 up, 22 in<br />pgmap v14727: 1408 pgs, 5 pools, 11977 MB data, 3183 objects<br />24729 MB used, 16361 GB / 16385 GB avail<br />1408 active+clean</p></blockquote>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-17578920707065988332016-11-16T16:42:00.000-06:002023-03-24T16:45:00.618-06:00Cloning a Ceph client auth key<p> <span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">I don't recall any reason to do this other than using the same user and auth key to authenticate in different Ceph clusters, like in a multi-backend solution, or just because things get messy when you are not using a default configuration.</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Sometimes, things get easy when we use the same user and auth key on both clusters for services to connect to, so let's see some background commands for managing users, keys, and permissions:</p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Create a new user and auth token (cinder client example)</span>:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">root@ceph-admin:~# </span>ceph auth get-or-create client.jerry<br />client.jerry<br />key: AQAZT05WoQuzJxAAX5BKxCbPf93CwihuHo27VQ==</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">So as you see the key is not a parameter, in a different server this will produce a completely different key.<br />Just to check, print the complete list of keys:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">root@ceph-admin:~# </span>ceph auth list<br />installed auth entries:</p><p style="margin: 1em 0px;">osd.0<br />key: AQCvCbtToC6MDhAATtuT70Sl+DymPCfDSsyV4w==<br />caps: [mon] allow profile osd<br />caps: [osd] allow *<br />osd.1<br />key: AQC4CbtTCFJBChAAVq5spj0ff4eHZICxIOVZeA==<br />caps: [mon] allow profile osd<br />caps: [osd] allow *<br />client.admin<br />key: AQBHCbtT6APDHhAA5W00cBchwkQjh3dkKsyPjw==<br />caps: [mds] allow<br />caps: [mon] allow *<br />caps: [osd] allow *<br />client.jerry<br />key: <span style="font-weight: bolder;">AQAZT05WoQuzJxAAX5BKxCbPf93CwihuHo27VQ==</span></p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Or print a user’s authentication key to standard output, execute the command in the following format<br /></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">ceph auth print-key {TYPE}.{ID}</p></blockquote><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">root@ceph-admin:~# </span>ceph auth print-key client.jerry<br />AQAZT05WoQuzJxAAX5BKxCbPf93CwihuHo27VQ==</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">To change this in order to match with others, we need to update their keys and/or their capabilities, the import command is for this, remember <span style="font-weight: bolder;">their keys and their capabilities will update on existing users and create new ones</span>, use the following format:<br /></p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;">ceph auth import -i /path/to/keyring</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">The keyring file needs to be in this format, if not, the command will not work and the part of the work, it will just hang.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">root@ceph-admin:~# </span>cat jerry.key<br />[client.jerry]<br />key = AQAMP01WS8i8ERAAPspjwMzUm4SL00n+WppM6A==</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Now we can update the auth key for the user jerry:</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">root@ceph-admin:~# </span>auth import -i ./jerry.key<br />imported keyring</p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">List again.</p><blockquote style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 40px;"><p style="margin: 1em 0px;"><span style="font-weight: bolder;">root@ceph-admin:~# </span>ceph auth list<br />installed auth entries:</p><p style="margin: 1em 0px;">osd.0<br />key: AQCvCbtToC6MDhAATtuT70Sl+DymPCfDSsyV4w==<br />caps: [mon] allow profile osd<br />caps: [osd] allow *<br />osd.1<br />key: AQC4CbtTCFJBChAAVq5spj0ff4eHZICxIOVZeA==<br />caps: [mon] allow profile osd<br />caps: [osd] allow *<br />client.admin<br />key: AQBHCbtT6APDHhAA5W00cBchwkQjh3dkKsyPjw==<br />caps: [mds] allow<br />caps: [mon] allow *<br />caps: [osd] allow *<br />client.jerry<br />key: <span style="font-weight: bolder;">AQAMP01WS8i8ERAAPspjwMzUm4SL00n+WppM6A==</span></p></blockquote><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">Done, I will continue posting these little helping tricks until the last post about multi-backend ceph is out.</p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-7593243294117063452016-03-18T22:52:00.001-06:002023-03-22T22:56:12.849-06:00Export instance from OpenStack with Ceph/rbd backend<p><span style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;">Suppose that you want to migrate an instance from different infrastructures or you want to hand over an instance information to a client, so you need to recover (export) the instance volumes information.</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;"><br /></span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Step 1</span>: Get the instance UUID.</p><pre style="background-color: white; color: #333333; font-family: monospace, monospace; font-size: 13.008px; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@ceph-admin:~#</span> nova list | grep InstanceToExport
| <span style="font-weight: bolder;">2bdda36c-f0dd-4fa5-bb8b-3df346b17002</span> | InstanceToExport | SHUTOFF | - | Shutdown | vlan8=192.168.255.53; vlan1837=10.20.37.7; vlan1829=10.20.23.53 |
</blockquote></pre><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br /></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">UUID from the instance is returned here: <span style="font-weight: bolder;">2bdda36c-f0dd-4fa5-bb8b-3df346b17002</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;"><br /></span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Step 2</span>: Get the volume UUID from the instance, using the UUID returned in step 1</p><pre style="background-color: white; color: #333333; font-family: monospace, monospace; font-size: 13.008px; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@ceph-admin:~#</span> cinder list | grep 2bdda36c-f0dd-4fa5-bb8b-3df346b17002
| <span style="font-weight: bolder;">fdb279c5-24bb-45d7-a86a-a33f4c285b5a</span> | in-use | None | 100 | None | true | 2bdda36c-f0dd-4fa5-bb8b-3df346b17002 |
</blockquote></pre><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br /></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">UUID from the volume is returned here: <span style="font-weight: bolder;">fdb279c5-24bb-45d7-a86a-a33f4c285b5a</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;"><br /></span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Step 3</span>: Search from the volume on the pool in ceph, in my case this volume is stored in the cinder-volumes pool</p><pre style="background-color: white; color: #333333; font-family: monospace, monospace; font-size: 13.008px; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@ceph-admin:~#</span> rbd --pool cinder-volumes ls | grep fdb279c5-24bb-45d7-a86a-a33f4c285b5a
volume-fdb279c5-24bb-45d7-a86a-a33f4c285b5a
</blockquote></pre><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><br /></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;">By this time you have the volume name in the pool: <span style="font-weight: bolder;">volume-fdb279c5-24bb-45d7-a86a-a33f4c285b5a</span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;"><br /></span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Step 4</span>: Export the volume.</p><pre style="background-color: white; color: #333333; font-family: monospace, monospace; font-size: 13.008px; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@ceph-admin:~#</span> rbd export cinder-volumes/volume-fdb279c5-24bb-45d7-a86a-a33f4c285b5a ./InstanceToExport.img
Exporting image: 100% complete...done.
</blockquote><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@ceph-admin:~#</span> ll -ltrh *.img
-rw-r--r-- 1 root root 100G Feb 17 17:09 InstanceToExport.img
</blockquote></pre><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;"><br /></span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Step 5</span>: Compress, so you can scp or rsync faster, this step is optional but highly recommended.</p><pre style="background-color: white; color: #333333; font-family: monospace, monospace; font-size: 13.008px; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"><br /></span></pre><pre style="background-color: white; color: #333333; font-family: monospace, monospace; font-size: 13.008px; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@ceph-admin:~#</span> gzip -9 InstanceToExport.img
<span style="font-weight: bolder;">root@ceph-admin:~#</span> ll *.gz
-rw-r--r-- 1 root root 1.2G Feb 17 18:02 InstanceToExport.img.gz
</blockquote></pre><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;"><br /></span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Step 6</span>: Checksum, to be sure that you don't have any problem copying</p><pre style="background-color: white; color: #333333; font-family: monospace, monospace; font-size: 13.008px; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@ceph-admin:~#</span> md5sum InstanceToExport.img >InstanceToExport.img.md5
<span style="font-weight: bolder;">root@ceph-admin:~#</span> md5sum InstanceToExport.img.gz >InstanceToExport.img.gz.md5
<span style="font-weight: bolder;">root@ceph-admin:~#</span> cat InstanceToExport.img.md5
5504cdf2261556135811fdd5787b33a5 InstanceToExport.img
<span style="font-weight: bolder;">root@ceph-admin:~#</span> cat InstanceToExport.img.gz.md5
8a76c28d404f44cc43872e69c9965cd2 InstanceToExport.img.gz
</blockquote></pre><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;"><br /></span></p><p style="background-color: white; color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"><span style="font-weight: bolder;">Note</span>: the <span style="font-weight: bolder;">md5sum InstanceToExport.img</span> is going to take a lot! in my volume (100G) like 20 minutes, omit it if you want.</p>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-70752489253245705172016-03-12T22:36:00.036-06:002023-03-22T22:44:54.366-06:00Testing juju environment inside LXC container<div class="dialog-off-canvas-main-canvas" data-off-canvas-main-canvas="" style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"><div class="layout-container" style="margin: 0px 2.5em;"><main class="page-content clearfix" role="main" style="margin-bottom: 80px;"><div class="region region-content"><article class="node node--type-article node--promoted node--view-mode-full" role="article"><div class="node__content"><div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p style="margin: 1em 0px; text-align: justify;">I think we pass the part about what juju is and how it works, so I'll post direct commands and configurations of how to get the environment working inside an LXC container, created just for juju, not the local configuration that creates an LXC container, in other words, out host server does not have any juju package.</p><p style="margin: 1em 0px;">Some links to read in case you need more info, or you can post a question.</p><ul style="list-style-image: none; margin: 0.25em 0px 0.25em 1.5em;"><li><a href="https://jujucharms.com/docs/stable/getting-started" style="color: #0074bd; text-decoration-line: none;" target="_blank">https://jujucharms.com/docs/stable/getting-started</a></li><li><a href="https://jujucharms.com/docs/1.24/commands" style="color: #0074bd; text-decoration-line: none;" target="_blank">https://jujucharms.com/docs/1.24/commands</a></li><li><a href="https://jujucharms.com/juju-gui/" style="color: #0074bd; text-decoration-line: none;" target="_blank">https://jujucharms.com/juju-gui/</a></li><li><a href="https://jujucharms.com/docs/1.22/howto-gui-management" style="color: #0074bd; text-decoration-line: none;" target="_blank">https://jujucharms.com/docs/1.22/howto-gui-management</a></li><li><a href="https://help.ubuntu.com/lts/clouddocs/en/Installing-Juju.html" style="color: #0074bd; text-decoration-line: none;" target="_blank">https://help.ubuntu.com/lts/clouddocs/en/Installing-Juju.html</a></li></ul><p style="margin: 1em 0px;">Guest environment:</p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@spyder:~#</span> cat /etc/issue
Ubuntu 15.10
<span style="font-weight: bolder;">root@spyder:~#</span> uname -a
Linux spyder 4.2.0-18-generic #22-Ubuntu SMP Fri Nov 6 18:25:50 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
<span style="font-weight: bolder;">root@spyder:~#</span> dpkg-query -W | grep lxc
liblxc1 1.1.5-0ubuntu5~ubuntu15.10.1~ppa1
lxc 1.1.5-0ubuntu5~ubuntu15.10.1~ppa1
lxc-templates 2.0.0~beta2-0ubuntu2~ubuntu15.10.1~ppa1
lxcfs 2.0.0~rc3-0ubuntu1~ubuntu15.10.1~ppa1
lxctl 0.3.1+debian-3
python3-lxc 1.1.5-0ubuntu5~ubuntu15.10.1~ppa1
</blockquote></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">In my host server, I have two lxcbr interfaces, but for the juju container I going to use lxcbr0, the container will have complete internet access but to access internal apps we are going to need DNAT iptables rules (at the end I'll post the iptables configuration).</p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@spyder:~#</span> ifconfig lxcbr0 | grep inet
inet addr:10.0.2.1 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::58b8:36ff:fe6e:4e57/64 Scope:Link
</blockquote></pre><p style="margin: 1em 0px;">Original lxc-ls output</p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@spyder:~#</span> lxc-ls --fancy
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
-------------------------------------------------------------------
ceph-admin RUNNING 10.0.2.11, 10.0.3.84 - - YES
ceph01 RUNNING 10.0.2.85, 10.0.3.85 - - YES
ceph02 RUNNING 10.0.2.103, 10.0.3.86 - - YES
ceph03 RUNNING 10.0.2.156, 10.0.3.87 - - YES
</blockquote></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">Now to the fun part, getting things working :)</p><p style="margin: 1em 0px;">First, create the juju container.</p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@spyder:~#</span> lxc-create -t download -n juju -- --dist ubuntu --release trusty --arch amd64
Using image from local cache
Unpacking the rootfs
---
You just created an Ubuntu container (release=trusty, arch=amd64, variant=default)
To enable sshd, run: apt-get install openssh-server
For security reason, container images ship without user accounts
and without a root password.
Use lxc-attach or chroot directly into the rootfs to set a root password
or create user accounts.
</blockquote></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">Start the container</p><p style="margin: 1em 0px;"><br /></p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@spyder:~#</span> lxc-start -n juju -d --logfile juju.log
<span style="font-weight: bolder;">root@spyder:~#</span> lxc-ls --fancy
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
-------------------------------------------------------------------
ceph-admin RUNNING 10.0.2.11, 10.0.3.84 - - YES
ceph01 RUNNING 10.0.2.85, 10.0.3.85 - - YES
ceph02 RUNNING 10.0.2.103, 10.0.3.86 - - YES
ceph03 RUNNING 10.0.2.156, 10.0.3.87 - - YES
juju RUNNING 10.0.2.110 - - YES
</blockquote></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">Now let's attach to the container (in this part you need to install openssh-server, set passwords to users, etc.)</p><p style="margin: 1em 0px;"><br /></p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@spyder:~#</span> lxc-attach --name juju
<span style="font-weight: bolder;">root@juju:~#</span> ip a
1: lo: <loopback> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
22: eth0@if23: <broadcast> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:16:3e:53:f1:f5 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.110/24 brd 10.0.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe53:f1f5/64 scope link
valid_lft forever preferred_lft forever
</broadcast></loopback></blockquote><loopback><broadcast></broadcast></loopback></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">To install Juju, you simply need to grab the latest juju-core package from the PPA:</p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"><br /></span></pre><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@juju:~#</span> apt-get install python-software-properties
<span style="font-weight: bolder;">root@juju:~#</span> apt-get install software-properties-common
<span style="font-weight: bolder;">root@juju:~#</span> add-apt-repository ppa:juju/stable
Stable release of Juju for Ubuntu 12.04 and above.
More info: https://launchpad.net/~juju/+archive/ubuntu/stable
Press [ENTER] to continue or ctrl-c to cancel adding it
gpg: keyring `/tmp/tmpyqs7twek/secring.gpg' created
gpg: keyring `/tmp/tmpyqs7twek/pubring.gpg' created
gpg: requesting key C8068B11 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpyqs7twek/trustdb.gpg: trustdb created
gpg: key C8068B11: public key "Launchpad Ensemble PPA" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
<span style="font-weight: bolder;">root@juju:~#</span> apt-get update
<span style="font-weight: bolder;">root@juju:~#</span> apt-get upgrade
<span style="font-weight: bolder;">root@juju:~#</span> apt-get install juju-quickstart juju-core
</blockquote></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">Juju needs to be configured to use your cloud provider. This is done via the following file:</p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><br /></pre><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><blockquote>$HOME/.juju/environments.yaml
</blockquote></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">Juju can automatically generate the file in this way:</p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"><br /></span></pre><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">ubuntu@juju:~$</span> juju generate-config
</blockquote></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">There are different types of clouds providers, check the environments.yaml for more info, the one important for us is the manual provider because we are going to deploy on manually on our same machine (LXC container in this case), so I deleted all the other information:</p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">ubuntu@juju:~$</span> cat /home/ubuntu/.juju/environments.yaml
default: manual
environments:
manual:
type: manual
# bootstrap-host holds the host name of the machine where the
# bootstrap machine agent will be started.
bootstrap-host: juju
# bootstrap-user specifies the user to authenticate as when
# connecting to the bootstrap machine. It defaults to
# the current user.
bootstrap-user: ubuntu
# storage-listen-ip specifies the IP address that the
# bootstrap machine's Juju storage server will listen
# on. By default, storage will be served on all
# network interfaces.
# storage-listen-ip:
# storage-port specifes the TCP port that the
# bootstrap machine's Juju storage server will listen
# on. It defaults to 8040
# storage-port: 8040
# Whether or not to refresh the list of available updates for an
# OS. The default option of true is recommended for use in
# production systems.
#
# enable-os-refresh-update: true
# Whether or not to perform OS upgrades when machines are
# provisioned. The default option of false is set so that Juju
# does not subsume any other way the system might be
# maintained.
#
# enable-os-upgrade: false
</blockquote></pre><p style="margin: 1em 0px; text-align: justify;"><br /></p><p style="margin: 1em 0px; text-align: justify;">The first step is to create a bootstrap environment. This is a cloud instance that Juju will use to deploy and manage services. It will be created according to the configuration you have provided, and your public SSH key will be uploaded automatically so that Juju can communicate securely with the bootstrap instance.</p><p style="margin: 1em 0px; text-align: justify;"><br /></p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">ubuntu@juju:~$</span> juju switch manual
manual -> manual
<span style="font-weight: bolder;">ubuntu@juju:~$</span> juju bootstrap
WARNING ignoring environments.yaml: using bootstrap config in file "/home/ubuntu/.juju/environments/manual.jenv"
Bootstrapping environment "manual"
Starting new instance for initial state server
Installing Juju agent on bootstrap instance
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Installing package: cloud-utils
Installing package: cloud-image-utils
Installing package: tmux
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools. tar.gz
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap agent installed
manual -> manual
Waiting for API to become available
Waiting for API to become available
Bootstrap complete
</blockquote></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">You can see that the bridge-utils package is installed, but inside an lxc container you are not going to use it and it can pass through an outside bridge to the juju container</p><p style="margin: 1em 0px;"><br /></p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@juju:~#</span> apt-get purge bridge-utils
</blockquote></pre><p style="margin: 1em 0px; text-align: justify;"><br /></p><p style="margin: 1em 0px; text-align: justify;">If you have any problem on the bootstrap delete conf files and start over, and I mean problems like the nasty <span style="font-weight: bolder;">“ERROR machine is already provisioned”</span> when the machine is not really provisioned.</p><p style="margin: 1em 0px; text-align: justify;"><br /></p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@juju:~#</span> apt-get purge lxc*
<span style="font-weight: bolder;">root@juju:~#</span> apt-get purge juju*
<span style="font-weight: bolder;">root@juju:~#</span> rm -rf /etc/init/juju*
<span style="font-weight: bolder;">root@juju:~#</span> rm -rf /var/lib/juju
</blockquote></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">If not, just continue, if everything is well right, you will see an output similar to this one, this means that the juju service is running on machine 0 (same LXC container).</p><p style="margin: 1em 0px;"><br /></p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">ubuntu@juju:~$</span> juju status
environment: manual
machines:
"0":
agent-state: started
agent-version: 1.25.3
dns-name: juju
instance-id: 'manual:'
series: trusty
hardware: arch=amd64 cpu-cores=2 mem=3000M
state-server-member-status: has-vote
services: {}
</blockquote></pre><p style="margin: 1em 0px; text-align: justify;"><br /></p><p style="margin: 1em 0px; text-align: justify;">Assuming it returns successfully, we can now deploy some services and explore the basic operations of Juju, next, you simply need to deploy our first charm (juju-gui) and expose it, this charm makes it easy to deploy a Juju GUI into an existing environment.</p><p style="margin: 1em 0px; text-align: justify;"><br /></p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span></pre><blockquote><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;">ubuntu@juju:~$</span> juju deploy juju-gui --to 0
<span style="font-weight: bolder;">ubuntu@juju:~$</span> juju expose juju-gui
........
........ after a couple of minutes, juju needs to download several packages and configure all, so better use "watch juju status", untill you see and output similar to this.
........
</pre><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;">ubuntu@juju:~$</span> juju status
environment: manual
machines:
"0":
agent-state: started
agent-version: 1.25.3
dns-name: juju
instance-id: 'manual:'
series: trusty
hardware: arch=amd64 cpu-cores=2 mem=3000M
state-server-member-status: has-vote
services:
juju-gui:
charm: cs:trusty/juju-gui-51
exposed: true
service-status:
current: unknown
since: 12 Mar 2016 09:12:45Z
units:
juju-gui/0:
workload-status:
current: unknown
since: 12 Mar 2016 09:12:45Z
agent-status:
current: idle
since: 12 Mar 2016 09:17:48Z
version: 1.25.3
agent-state: started
agent-version: 1.25.3
machine: "0"
open-ports:
- 80/tcp
- 443/tcp
public-address: juju
</pre></blockquote><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"></pre><p style="margin: 1em 0px; text-align: justify;"><br /></p><p style="margin: 1em 0px; text-align: justify;">Now the juju-gui is installed, configured, and exposed over ports 80 and 443, but remember, this is inside the LXC container, so we can't access the GUI unless we NAT some ports from our host server.</p><p style="margin: 1em 0px; text-align: justify;"><br /></p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">root@spyder:~#</span> iptables -t nat -A PREROUTING -p tcp -d 10.0.1.139 --dport 443 -j DNAT --to-destination 10.0.2.110:443
<span style="font-weight: bolder;">root@spyder:~#</span> iptables -t nat -A PREROUTING -p tcp -d 10.0.1.139 --dport 80 -j DNAT --to-destination 10.0.2.110:80
</blockquote></pre><p style="margin: 1em 0px;"><br /></p><p style="margin: 1em 0px;">And boom!!!! now we can access juju-gui, the login info is on this file:</p><p style="margin: 1em 0px;"><br /></p><pre style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><span style="font-weight: bolder;"></span><blockquote><span style="font-weight: bolder;">ubuntu@juju:~$</span> cat .juju/environments/manual.jenv
user: admin
password: 0d4e465c15d5880d0c348a921489a9f1
.......</blockquote></pre></div></div></article></div></main></div></div>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-68533332508196115802016-03-10T22:13:00.005-06:002023-03-22T22:14:36.818-06:00Cinder Volume Transfer<div class="dialog-off-canvas-main-canvas" data-off-canvas-main-canvas="" style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px;"><div class="layout-container" style="margin: 0px 2.5em;"><main class="page-content clearfix" role="main" style="margin-bottom: 80px;"><div class="region region-content"><article class="node node--type-article node--promoted node--view-mode-full" role="article"><div class="node__content"><div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p style="margin: 1em 0px;">Let's assume you want to change ownership of volume from Tenant_A to Tenant_B.</p><p style="margin: 1em 0px;">Step 1: Tenant A will initiate an Ownership Transfer which will enable another tenant to take ownership of it.</p><pre class="wiki" style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><blockquote>$ source openrc Tenant_A Tenant_A
$ cinder transfer-create [volume_id]
</blockquote></pre><p style="margin: 1em 0px;"><span style="font-weight: bolder;">An Authentication Key and a Transfer ID are returned here.</span></p><p style="margin: 1em 0px;">Step 2: Tenant B needs to accept the Transfer using the Transfer ID and The Authentication Key generated above.</p><pre class="wiki" style="font-family: monospace, monospace; font-size: 1em; margin-bottom: 0.5em; margin-top: 0.5em; white-space: pre-wrap;"><blockquote>$ source openrc Tenant_B Tenant_B
$ cinder transfer-accept [transfer_id] [auth_key]
</blockquote></pre><p style="margin: 1em 0px;"><span style="font-weight: bolder;">You should now see that volume associated with Tenant_B</span></p></div></div></article></div></main></div></div>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.comtag:blogger.com,1999:blog-91251585743147491.post-11772835254759584702015-03-12T22:06:00.028-06:002023-03-24T17:16:09.751-06:00The real problem behind highly transactional applications<div class="dialog-off-canvas-main-canvas" data-off-canvas-main-canvas=""><div class="layout-container" style="margin: 0px 2.5em;"><main class="page-content clearfix" role="main" style="margin-bottom: 80px;"><div class="region region-content"><article class="node node--type-article node--promoted node--view-mode-full" role="article"><div class="node__content"><div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">An architecture trying to respond to at least 10000 concurrent connections per second, is trying to solve the C10K problem, even if this is so last decade is still breaking servers, architectures, and configurations, giving sysadmins real headaches and not always because of real connections, also for basic DDoS attacks (pretty much is the same concept: lots and lots of new connections to the same service).</p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">Today, because of the need of connecting and sharing resources across infrastructures and also the need to implement high availability in solutions many companies have implemented SOA or multi-layer solutions when these solutions can become handy, it could also be a problem if are not implemented in the correct way: without the proper testing set, and sometimes people don't even know it if the architecture implemented is going to respond in the correct way or even the way that the developers team are planning. this problem does not only affect to wrong configured architectures but also solutions not properly planned to grow.</p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">The problem usually is errors in coding and validation on every layer of the application solution; proprietary code, web server, application server, DBMS, and so on, if applications were coded properly security and bug-hunting guys would be unemployed by now.</p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">So what are you going to see in a highly transactional server with a misconfiguration problem?</p><ul style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; list-style-image: none; margin: 0.25em 0px 0.25em 1.5em;"><li style="text-align: justify;">Lots of TIME_WAIT connections.</li><li style="text-align: justify;">Lots of CLOSE_WAIT connections.</li><li style="text-align: justify;">Possibly memory problems.</li><li style="text-align: justify;">Possibly the system swapping.</li><li style="text-align: justify;">Really Slow server.</li><li style="text-align: justify;">Many timeouts in the application log.</li><li style="text-align: justify;">The application became unreachable.</li><li style="text-align: justify;">We can't create new connections to the server, even ssh ones.</li><li style="text-align: justify;">... Worst case scenario, dead servers.</li></ul><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">But service restart, reboot and kill will not solve all the problems, nor the operating system or the kernel are there to solve all the problems, the kernel work is to handle the control plane and in a general and multipurpose way, if you take only the kernel tuning approach, the kernel is going to be part of the problem, and you are going to be far far away to solve the problem.</p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">The kernel has a known way to work and knows O(n^2) complexity, with every new connection the kernel has to walk down all the current processes to figure out which thread should handle the packet or if we talk of connection polls the process is the same, each packet had to walk a list of sockets.</p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><br /></p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><center><div style="text-align: justify;"><span face=""Open Sans", sans-serif" style="color: #333333; font-size: 13.008px;"><br /></span></div><span face=""Open Sans", sans-serif" style="color: #333333; font-size: 13.008px; height: auto;"><div class="separator" style="clear: both; text-align: center;"><a href="https://us.mirrors.headup.ws/images/OS-structure2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="203" data-original-width="800" height="159" src="https://us.mirrors.headup.ws/images/OS-structure2.png" width="552" /></a></div><br /><div style="text-align: center;"><br /></div></span><span face=""Open Sans", sans-serif" style="color: #333333; font-size: 13.008px;"><div style="text-align: center;"><span style="font-size: 13.008px; text-align: left;">Hight level Kernel diagram: layers and intercommunication </span><span style="font-size: 13.008px; font-weight: bolder; text-align: left;">(1)</span><span style="font-size: 13.008px; text-align: left;">.</span></div></span></center><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><br /></p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><br /></p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">Even if you take the complete tuning approach, maybe the application is going to work, but not always, you only are going to get stability, but not the real solution, the correct way to handle and solve the C10K problem, even more, C10M is letting the kernel solve de control plane and applications handle the data plane and/or write software to bypass the stack, such as DPDK <span style="font-weight: bolder;">(2)</span>, this is pretty much like if we're talking about an exokernel <span style="font-weight: bolder;">(3)</span>, using an end-to-end principle.</p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><br /></p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><center><div style="text-align: justify;"><span face=""Open Sans", sans-serif" style="color: #333333; font-size: 13.008px;"><br /></span></div><span face=""Open Sans", sans-serif" style="color: #333333; font-size: 13.008px; height: auto;"><div class="separator" style="clear: both; text-align: center;"><a href="https://us.mirrors.headup.ws/images/ExokernelRevisedEnglish.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="325" data-original-width="457" height="325" src="https://us.mirrors.headup.ws/images/ExokernelRevisedEnglish.png" width="457" /></a></div><br /><div style="text-align: center;"><br /></div></span><span face=""Open Sans", sans-serif" style="color: #333333; font-size: 13.008px;"><div style="text-align: center;"><span style="font-size: 13.008px; text-align: left;">Common Kernel V/S ExoKernel </span><span style="font-size: 13.008px; font-weight: bolder; text-align: left;">(3)</span><span style="font-size: 13.008px; text-align: left;">.</span></div></span></center><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><br /></p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><br /></p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">To build usable and scalable applications to support 10 million concurrent connections per second (and more), we need to solve other kinds of problems first.</p><ul style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; list-style-image: none; margin: 0.25em 0px 0.25em 1.5em;"><li style="text-align: justify;">Packet scalability.</li><li style="text-align: justify;">Multi-core scalability.</li><li style="text-align: justify;">Memory scalability.</li></ul><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><i>So the real problem is.... <span style="font-weight: bolder;">knowledge</span></i>, lots of developers know how to code client/server applications, but less than 50% of them know how the TCP/IP or TTCP/IP works, and how to use MP libraries, I understand this is not an easy task to accomplish, but we really need to start working on that, with every performance problem we also need to start looking in the code and software architecture searching for scalability errors, not always will have site reliability engineers to help our application to be super reliable, super fast, all the time, even if we have these guys to help us, the solution can be found many iterations behind before the system starts losing points of our precious 99.99…99</p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">And what if, we can correct coding errors fast enough or we can’t (in the case of proprietary software): tuning, will always be the answer, but like I said, tune all the layers, not only the kernel:</p><ul style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; list-style-image: none; margin: 0.25em 0px 0.25em 1.5em;"><li style="text-align: justify;">Tune for aggressive network throughput.</li><li style="text-align: justify;">Tune timeouts.</li><li style="text-align: justify;">Tune the socket parameters.</li><li style="text-align: justify;">Tune shared filesystems.</li><li style="text-align: justify;">Tune the schedulers.</li><li style="text-align: justify;">Tune the complete architecture.</li><li style="text-align: justify;">….</li></ul><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">There are many layers before can reach the kernel, and even if you want to tune the kernel you need to understand how the application works, communicate and use internal and external applications, libraries, and utilities.</p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><br /></p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px;"></p><center><div style="text-align: justify;"><span face=""Open Sans", sans-serif" style="color: #333333; font-size: 13.008px;"><br /></span></div><span face=""Open Sans", sans-serif" style="color: #333333; font-size: 13.008px; height: auto;"><div class="separator" style="clear: both; text-align: center;"><a href="https://us.mirrors.headup.ws/images/CommonApplicationArchitecture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="475" data-original-width="504" height="475" src="https://us.mirrors.headup.ws/images/CommonApplicationArchitecture.png" width="504" /></a></div><br /><div style="text-align: center;"><br /></div></span><span face=""Open Sans", sans-serif" style="color: #333333; font-size: 13.008px;"><div style="text-align: center;"><span style="font-size: 13.008px; text-align: left;">Common multi-layer software architecture </span><span style="font-size: 13.008px; font-weight: bolder; text-align: left;">(4)</span><span style="font-size: 13.008px; text-align: left;">.</span></div></span></center><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><br /></p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;"><br /></p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">In common transactional architectures, tuning will work like a tourniquet in a bullet wounded, probably saving a life but In highly transactional applications, tuning is just to help the system, not to solve problems and your application will die slowly and painfully.</p><p style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 1em 0px; text-align: justify;">References:</p><ol style="color: #333333; font-family: "Open Sans", sans-serif; font-size: 13.008px; margin: 0.25em 0px 0.25em 2em; padding: 0px;"><li style="text-align: justify;"><a href="https://en.wikipedia.org/wiki/Monolithic_kernel" style="color: #0074bd; text-decoration-line: none;">https://en.wikipedia.org/wiki/Monolithic_kernel</a></li><li style="text-align: justify;"><a href="http://dpdk.org/" style="color: #0074bd; text-decoration-line: none;">http://dpdk.org/</a></li><li style="text-align: justify;"><a href="https://en.wikipedia.org/wiki/Exokernel" style="color: #0074bd; text-decoration-line: none;">https://en.wikipedia.org/wiki/Exokernel</a></li><li style="text-align: justify;"><a href="http://www.guidanceshare.com/" style="color: #0074bd; text-decoration-line: none;">http://www.guidanceshare.com</a></li></ol></div></div></article></div></main></div></div>alvaro.sotohttp://www.blogger.com/profile/00465925061882896036noreply@blogger.com