Scripting Bright Cluster Manager 9.0 with Python

It has been more than 6 years since the previous post about using the Python API to script Bright Cluster Manager (CM). Time for an update.

I have to do the same as before: change the “category” on a whole bunch of nodes.

NB the Developer Manual has some typos, where it makes it look like you can specify categories as strings of their names, e.g. cluster.get_by_type('Node')

Building Ganglia Monitor Core 3.6.1 on RHEL 7 - "Installed (but unpackaged) file(s) found" error

Install a bunch of prerequisites, some from EPEL:

* libconfuse

* libart_lgpl-devel

Download source: monitor-core-3.6.1.tar.gz

Expand the tarball: tar xf monitor-core-3.6.1.tar.gz  That creates the directory monitor-core-3.6.1

Enter that directory and run the bootstrap script: ./bootstrap.sh

That generates the configure script. 

Run the configure script: ./configure

That generates the SPEC file: ganglia.spec

Make a tar ball with the appropriate name:

    cd ..

    mv monitor-core-3.6.1 ganglia-3.6.1

    tar zcf ganglia-3.6.1.tar.gz ganglia-3.6.1

Build with rpmbuild -ta ganglia-3.6.1.tar.gz

Will probably get RPM build errors:

    bogus date in %changelog: Thu Mar 28 2008 Brad Nicholes <bnicholes@novell.com>

    bogus date in %changelog: Wed Jul 10 2007 Bernard Li <bernard@vanhpc.org>

    bogus date in %changelog: Wed Jul 3 2007 Brad Nicholes <bnicholes@novell.com>

    bogus date in %changelog: Wed Jun 14 2007 Brad Nicholes <bnicholes@novell.com>

    bogus date in %changelog: Fri Feb 25 2006 Bernard Li <bli@bcgsc.ca>

    Installed (but unpackaged) file(s) found:

   /usr/lib/systemd/system/gmetad.service

   /usr/lib/systemd/system/gmond.service

Modify the SPEC file ganglia.spec: add line in "%files gmetad" section:

    /usr/lib/systemd/system/gmetad.service

and in "%files gmond" section:

    /usr/lib/systemd/system/gmond.service

Make a new tar ball containing the fixed SPEC file (delete the old one first):

    rm ganglia-3.6.1.tar.gz

    tar zcf ganglia-3.6.1.tar.gz ganglia-3.6.1

Then build with: rpmbuild -ta ganglia-3.6.1.tar.gz

RPMs should be in: ~/rpmbuild/RPMS/x86_64

Dealing with Excel CSV/TSV files in UTF-16LE encoding, and an invisible character

Motivation: I am trying to read a TSV file produced by Blackboard courseware. I am trying to do so using Python's built-in csv library. I am using a simple method:

import csv 

with open('myfile.tsv', 'r', encoding='utf-16le') as cf:
    cr = csv.DictReader(cf, dialect='excel-tab')
    print(cr.fieldnames)   

    for row in cr:       
        print(row)

What I got for the fieldnames was:

['\ufeff"Last Name"', 'First Name']

I.e. the "Last Name" field got munged with \ufeff or U+FEFF in normal Unicode notation.

and the rows looked like:

OrderedDict([('\ufeff"Last Name"', 'Doe'), ('First Name', 'Alice')])

Just using vim to look at the file, there seemed to be nothing weird about the first line which contains the column names. This is a ZERO WIDTH NO-BREAK SPACE character used as a Byte Order Mark (BOM). It allows reading processes, e.g. file(1)/magic(5), to figure out the byte order of the file.

But it messes up the csv.DictReader parsing of the field names.

Turns out, there is an easy fix. You can just modify the field names directly:

cr = csv.DictReader(cf, dialect='excel-tab')
cr.fieldnames[0] = 'Last Name'

And the output is fixed, too:

OrderedDict([('Last Name', 'Doe'), ('First Name', 'Alice')])

RHEL7 on VirtualBox graphics controller

I have a RHEL7 VM on VirtualBox 6.0 which crashed right after a full update to RHEL 7.7, including kernel. Just typed "startx" (I usually make VMs boot into console), and it crashed immediately with a "Guru Meditation" from VirtualBox.

Taking a stab, I changed the Graphics Controller for the VM from VMSVGA to VBoxVGA, and it launched the GUI just fine.

Even more about SSSD + PAM + LDAP -- password is still expired even right after being changed by user

This keeps coming back to haunt me, partly because of patchy and disparate documentation, and partly because I do not have a rock-solid understanding of all the details of SSSD + PAM + LDAP. (Previous post.)

This is for RHEL6.

Here is the issue: my users kept running into the instance when upon logging in, they were shown:

WARNING: Your password has expired.
You must change your password now and login again!
Changing password for foouser.
Current password:

And then it automatically logs you out, which is expected behavior.

However, when they login again (with the password that they just set), they are again presented with the same password expiration warning. This repeats ad infinitum.

When I check the OpenLDAP server, and ldapsearch for the user record, it does show that the password was changed by that user on the correct date.

The key bit that I seem to have missed: a setting in /etc/pam_ldap.conf You have to set the secure LDAP URI since SSSD password transmissions must be encrypted.

uri ldaps://10.9.8.7/

This should match the URI specified in /etc/openldap/ldap.conf

URI ldaps://10.9.8.7/

And the setting in /etc/sssd/sssd.conf

[domain/default]
...
ldap_uri = ldaps://10.9.8.7/
...

And that fixed it.

While you are at it, you might as well specify SHA512 for the hash in /etc/pam_ldap.conf

pam_password sha512

I RTFMed: "sha512" is not an option for pam_password. This is to hash the password locally, before passing on to the LDAP server. The default is "clear", i.e. transmit the password in the clear to the LDAP server, and assume the LDAP server will hash if necessary. Another option is "crypt" which uses crypt(3).

pam_password crypt

However, there does not seem to be a way to specify which hash algorithm is to be used.

I do not think this is a big issue because the connection to the LDAP server is encrypted, any way.

Why was this a surprise? Well, because in /etc/nsswitch.conf we specified sss as the source for the passwd, shadow, and group name services:

passwd:     files sss
shadow:     files sss
group:      files sss

I.e., everything should be mediated through SSSD, and the SSSD config does have the correct URI.

U2F USB key (Yubikey) for 2-factor authentication and Linux authentication

I just bought a pair of Yubikey U2F (Universal 2-Factor) devices (the 5 NFC model, because of the claims that it would work with iPhones). Mostly because I got tired of pulling out my phone, finding the authenticator app, searching for the entry for the appropriate website, and then typing in the number.

I'll get to the iPhone stuff at the end.

But first, using the Yubikey for the second factor works for only a few websites. Also, it depends on your web browser: I tested Chrome (on Linux, macOS, and Chromebook), and Firefox (on Linux and macOS). Chrome and Firefox can deal with reading a U2F key via USB just fine.

Yubico has clear instructions for how to set the keys up: https://www.yubico.com/setup

Among sites which accept U2F hardware keys are Facebook, Google, GitHub, GitLab,  Dropbox, and Twitter (though Twitter does not support multiple U2F keys, which sucks if you lose a key). You browse to the site as usual, type in your password, and it will prompt you to plug in your U2F key and tap the flashing bit with a gold contact sensor, and you're in.

For using the Yubikey as a U2F in Linux, to authenticate for logging in, unlocking the screensaver, and sudo, you will have to install Yubico's U2F PAM module: https://github.com/Yubico/yubico-pam There is more detailed documentation geared towards developers here: https://developers.yubico.com/yubico-pam/ The PAM module works fine in Ubuntu 19.10 Eoan Ermine.

And it works great: plug the Yubikey in first, type in your password and hit Enter, the key starts flashing, touch the flashing bit, and you are in.

On the downside, I would not use this on a server where you need to do management remotely, since you would not be able to plug in a U2F key on an SSH connection.

As for using NFC and iOS: it does not work like I expected it to, nor how the Yubico website led me to expect. If you tap the Yubikey to the iPhone, it will pop up an alert, which if you tap, will open Safari on a "validation" web page hosted at yubico.com.

The websites which work within Chrome and Firefox on a computer (Google, GitHub, etc) do not seem to have a way to read the Yubikey via NFC on iPhone. There is a Lightning + USB-C key (the 5 Ci) but it's expensive ($70 ea.) and I do not know for sure if it will work, since the Google and GitHub mobile websites viewed on iPhone and Android do not even present the option for using U2F keys.

So, at this point, I feel I should have just bought the cheaper non-NFC, and I would have been at the same point.

UPDATE 1 If you use KeePassXC for storing passwords, it can be configured to require a YubiKey. This uses the "challenge-response" feature, which has to be manually set up using the YubiKey Personalization Tool (also available at GitHub). Yubico has a video walkthrough here: https://www.yubico.com/products/services-software/personalization-tools/challenge-response/

OpenLDAP annoyance

The OpenLDAP 2.4 administrator's manual is missing a section:

https://www.openldap.org/devel/admin/monitoringslapd.html

18.1. Monitor configuration via cn=config(5)
This section has yet to be written.