As Rusty Russell once said, if you want to get smarter, the thing to do is to hang out with smart people. An email-based workflow lets developers hang out with a project's smart people, making them all smarter. Greg wants Linux to last a long time, so wants to see the kernel project use tools that help to bring in new developers. Email, for all its flaws, is still better than anything else in that regard.
How-to's and technical news about Linux and open computing, with a sprinkling of Python.
2017-09-30
Why kernel development still uses email
Good post on why email is ideal for kernel development process:
2017-09-22
Shorewall setup for VirtualBox host-only interface
VirtualBox has a networking mode called "host-only" which allows guests to communicate with each other, and the host to communicate with the guests.
To do this, a host-only network (interface) must be defined on the host. It can be done via GUI:
or via the commandline (needs sudo because this creates a new network interface on the host):
$ sudo vboxmanage hostonlyif create
This creates a host-only virtual interface on the host, named vboxnetN (N starts at 0 and increments for each new one):
$ ip addr list
...
12: vboxnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether ...
inet 192.168.56.1/24 brd 192.168.56.255 scope global vboxnet0
inet6 fe80::800:27ff:fe00:0/64 scope link
valid_lft forever preferred_lft forever
# /etc/shorewall/zones
#ZONE TYPE OPTIONS IN OUT
# OPTIONS OPTIONS
vh ipv4
To do this, a host-only network (interface) must be defined on the host. It can be done via GUI:
or via the commandline (needs sudo because this creates a new network interface on the host):
$ sudo vboxmanage hostonlyif create
This creates a host-only virtual interface on the host, named vboxnetN (N starts at 0 and increments for each new one):
$ ip addr list
...
12: vboxnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether ...
inet 192.168.56.1/24 brd 192.168.56.255 scope global vboxnet0
inet6 fe80::800:27ff:fe00:0/64 scope link
valid_lft forever preferred_lft forever
There are three things to do in Shorewall: define a zone, place the host-only interface into that zone, and write a rule.
In /etc/shorewall/zones define the new zone:
#ZONE TYPE OPTIONS IN OUT
# OPTIONS OPTIONS
vh ipv4
In /etc/shorewall/interfaces put the host-only interface vboxnet0 in that zone:
# /etc/shorewall/interfaces
#ZONE INTERFACE BROADCAST OPTIONS
vh vboxnet0 detect dhcp
And finally, in /etc/shorewall/rules allow all traffic in the vh zone:
# /etc/shorewall/rules
ACCEPT vh:192.168.56.0/24 fw all
On the guest, create a new adapter, and either use DHCP or assign it a static IP in 192.168.56.0/24 (excluding 192.168.56.1, which is the host's IP address). Attach the adapter to the Host-only Adapter:
Or use the command line:
$ vboxmanage modifyvm myguest --nic2 hostonly
Or use the command line:
$ vboxmanage modifyvm myguest --nic2 hostonly
Restart the shorewall service, and that should do it. Test it out by ssh'ing into the guest from the host.
2017-09-09
Proposed fix for duplicity Azure backend breakage
At work, I just got a Microsoft Azure Cool Blob Storage allocation for doing off-site backups. The Python-based duplicity software is supposed to be able to use Azure Blob storage as a backend. It does this by using the azure-storage Python module provided by Microsoft.
Unfortunately, a recent update of azure-storage broke duplicity. The fix was not to hard to implement; mostly minor changes in class names, and one simplification in querying blob properties. It took me a few hours to make a fix, and I just submitted my changes as a merge request to duplicity.The proposed merge can be found at Launchpad.
UPDATE Unfortunately, I made a mistake and made my changes against the 0.7.14 release rather than trunk. It looks like there is already a lot of work in trunk to deal with the current azure-storage version. So, I withdrew the merge request. I'll work from the 0.8 series branch, instead. Currently, it looks like 0.8 all works as is.
Unfortunately, a recent update of azure-storage broke duplicity. The fix was not to hard to implement; mostly minor changes in class names, and one simplification in querying blob properties. It took me a few hours to make a fix, and I just submitted my changes as a merge request to duplicity.
UPDATE Unfortunately, I made a mistake and made my changes against the 0.7.14 release rather than trunk. It looks like there is already a lot of work in trunk to deal with the current azure-storage version. So, I withdrew the merge request. I'll work from the 0.8 series branch, instead. Currently, it looks like 0.8 all works as is.
2016-11-10
Linux daemon using Python daemon with PID file and logging
The python-daemon package (PyPI listing, Pagure repo) is very useful. However, I feel it has suffered a bit from sparse documentation, and the inclusion of a "runner" example, which is in the process of being deprecated as of 2 weeks ago (2016-10-26).
There are several questions about it on StackOverflow, going back a few years: 2009, 2011, 2012, and 2015. Some refer to the included runner.py as an example, which is being deprecated.
So, I decided to figure it out myself. I wanted to use the PID lockfile mechanism provided by python-daemon, and also the Python logging module. The inline documentation for python-daemon mention the files_preserve parameter, a list of file handles which should be held open when the daemon process is forked off. However, there wasn't an explicit example, and one StackOverflow solution for logging under python-daemon mentions that the file handle for logging objects may not be obvious:
There are several questions about it on StackOverflow, going back a few years: 2009, 2011, 2012, and 2015. Some refer to the included runner.py as an example, which is being deprecated.
So, I decided to figure it out myself. I wanted to use the PID lockfile mechanism provided by python-daemon, and also the Python logging module. The inline documentation for python-daemon mention the files_preserve parameter, a list of file handles which should be held open when the daemon process is forked off. However, there wasn't an explicit example, and one StackOverflow solution for logging under python-daemon mentions that the file handle for logging objects may not be obvious:
- for a StreamHandler, it's logging.root.handlers[0].stream.fileno()
- for a SyslogHandler, it's logging.root.handlers[1].socket.fileno()
After a bunch of experiments, I think I have sorted it out to my own satisfaction. My example code is in GitHub: prehensilecode/python-daemon-example. It also has a SysV init script.
The daemon itself is straigtforward, doing nothing but logging timestamps to the logfile. The full code is pasted here:
#!/usr/bin/env python3.5 import sys import os import time import argparse import logging import daemon from daemon import pidfile debug_p = False def do_something(logf): ### This does the "work" of the daemon logger = logging.getLogger('eg_daemon') logger.setLevel(logging.INFO) fh = logging.FileHandler(logf) fh.setLevel(logging.INFO) formatstr = '%(asctime)s - %(name)s - %(levelname)s - %(message)s' formatter = logging.Formatter(formatstr) fh.setFormatter(formatter) logger.addHandler(fh) while True: logger.debug("this is an DEBUG message") logger.info("this is an INFO message") logger.error("this is an ERROR message") time.sleep(5) def start_daemon(pidf, logf): ### This launches the daemon in its context global debug_p if debug_p: print("eg_daemon: entered run()") print("eg_daemon: pidf = {} logf = {}".format(pidf, logf)) print("eg_daemon: about to start daemonization") ### XXX pidfile is a context with daemon.DaemonContext( working_directory='/var/lib/eg_daemon', umask=0o002, pidfile=pidfile.TimeoutPIDLockFile(pidf), ) as context: do_something(logf) if __name__ == "__main__": parser = argparse.ArgumentParser(description="Example daemon in Python") parser.add_argument('-p', '--pid-file', default='/var/run/eg_daemon.pid') parser.add_argument('-l', '--log-file', default='/var/log/eg_daemon.log') args = parser.parse_args() start_daemon(pidf=args.pid_file, logf=args.log_file)
2016-11-08
scikit-learn with shared CBLAS and BLAS
If you have your own copies of BLAS and CBLAS installed as shared libraries, the default build of scikit-learn may end up not finding libblas.so which libcblas.so depends on.
You may, when doing "from sklearn import svm", get an error like:
from . import libsvm, liblinearImportError: /usr/local/blas/lib64/libcblas.so: undefined symbol: cgemv_
To fix it, modify the private _build_utils module:
scikit-learn-0.xx/sklearn/_build_utils/__init__.py
--- __init__.py.orig 2016-11-08 16:19:49.920389034 -0500
+++ __init__.py 2016-11-08 15:58:42.456085829 -0500
@@ -27,7 +27,7 @@
blas_info = get_info('blas_opt', 0)
if (not blas_info) or atlas_not_found(blas_info):
- cblas_libs = ['cblas']
+ cblas_libs = ['cblas', 'blas']
blas_info.pop('libraries', None)
else:
cblas_libs = blas_info.pop('libraries', [])
You may, when doing "from sklearn import svm", get an error like:
from . import libsvm, liblinearImportError: /usr/local/blas/lib64/libcblas.so: undefined symbol: cgemv_
To fix it, modify the private _build_utils module:
scikit-learn-0.xx/sklearn/_build_utils/__init__.py
--- __init__.py.orig 2016-11-08 16:19:49.920389034 -0500
+++ __init__.py 2016-11-08 15:58:42.456085829 -0500
@@ -27,7 +27,7 @@
blas_info = get_info('blas_opt', 0)
if (not blas_info) or atlas_not_found(blas_info):
- cblas_libs = ['cblas']
+ cblas_libs = ['cblas', 'blas']
blas_info.pop('libraries', None)
else:
cblas_libs = blas_info.pop('libraries', [])
2016-11-07
Optimized zlib
I wasn't aware of optimized versions of zlib, the free patent-unencumbered compression library, until today. I ran across Juho Snellman's comparison benchmarks of vanilla zlib, CloudFlare zlib, Intel zlib, and zlib-ng. The upshot is that CloudFlare's optimizations seem to be the best performing. And its decompression times were 75% of the vanilla version, while Intel and zlib-ng ran at 98% and 99% respectively. This would be a clear win for read-intensive workflows, such as some bioinformatics workflows.
Read more about how CloudFlare was contacted by the Institute of Cancer Research in London to help improve zlib at this blog post by Vlad Krasnov. Intel has an overview of zlib in its Intel Performance Primitives (IPP) product. And this is zlib-ng's GitHub repo.
Read more about how CloudFlare was contacted by the Institute of Cancer Research in London to help improve zlib at this blog post by Vlad Krasnov. Intel has an overview of zlib in its Intel Performance Primitives (IPP) product. And this is zlib-ng's GitHub repo.
Subscribe to:
Posts (Atom)