VCT Labs

GNU screen cheatsheet: text-console session manager

2018-04-18T00:00:00-07:00

cheatsheet for GNU

GNU screen

Multi-user sessions

Suppose user1st and user2nd are the login names of the would-be collaborators on a single computer where the screen session

Commands for user1st who starts the screen session

screen
<hit a key to dismiss intro screen>
<ctrl-a> :multiuser on
<ctrl-a> :addacl user2nd

Note the space between ctrl-a keystroke and : is not meant to be typed; just here for readability (makes the : less likely to be overlooked)

Command for user2nd to join the same screen session

screen -rx user1st/

Scrollback

Start scrollback mode

<ctrl-a> <esc>

Note if you are already in scrollback mode when you do this sequence, you will receive the cryptic status message "Must be on a window layer" but it doesn’t hurt anything (you will still be in scrollback mode as intended).

Navigation

Can use arrow keys, page-up/page-down keys, vi-style navigation, and/or emacs-style navigation. This includes navigation by searching for literal strings, with both vi-style / (forwards) ? (backwards) and emacs-style <ctrl-s key-mappings supported.

End scrollback mode

<esc>

You will see status message "Copy mode aborted"

Logging

Start logging to a file /tmp/scroll-session.txt

<ctrl-a> :logfile /tmp/scroll-session.txt
<ctrl-a> :log on

You will see status "Creating logfile /tmp/scroll-session.txt" if the file doesn’t exist; status "Appending to logfile /tmp/scroll-session.txt" if it already does exist.

End logging

<ctrl-a> :log off

The resulting status message "Logfile /tmp/scroll-session.txt closed" will remind you where your logfile is located.

Embedded-board Serial Console Cheatsheet

2015-12-24T00:00:00-08:00

Conventions

TX/RX is labeled from perspective of development host (opposite of target board), thus called "host TX" and "host RX"
unless otherwise noted, software serial configuration is 15200 baud, 8N1, no flow control

Adafruit USB-Serial Cable

Adafruit PL2303 3.3V serial cable https://www.adafruit.com/product/954

GND BLACK
host TX GREEN
host RX WHITE

Minnowboard Max

Serial port is single-row right-angle 6-pin connector J4

Schematic: http://wiki.minnowboard.org/images/6/6d/MinnowBoard_Max_RevA2_sch.pdf

Numbering below starts from the left while holding board with serial port along top edge, so Pin 1 is closest to SATA connector

http://wiki.minnowboard.org/MinnowBoard_MAX#4-Wire_Serial_Console

Pin #	Role	Adafruit color
Pin 1	GND	BLACK
Pin 4	host TX	GREEN
Pin 5	host RX	WHITE

Note that these are not consecutive: pins 2, 3, and 6 should be empty. This is standard arrangement for boards compatible with an FTDI serial cable.

Beaglebone Black

Serial port is single-row 6-pin connector J1 running along the inner side of one of the expansion sockets for "cape" daughtercards

Schematic: https://github.com/CircuitCo/BeagleBone-Black/blob/master/BBB_SCH.pdf?raw=true

Numbering starts from the left while holding board with the serial port towards the bottom, so Pin 1 is the one closest to 5V power connector

http://dave.cheney.net/2013/09/22/two-point-five-ways-to-access-the-serial-console-on-your-beaglebone-black note text accompanying picture uses TX/RX from target perspective

Pin #	Role	Adafruit color
Pin 1	GND	BLACK
Pin 4	host TX	GREEN
Pin 5	host RX	WHITE

Note that these are not consecutive: pins 2, 3, and 6 should be empty. This is standard arrangement for boards compatible with an FTDI serial cable.

Raspberry Pi

Serial port is on double-row 26-pin connector J8

https://www.raspberrypi.org/documentation/hardware/raspberrypi/schematics/README.md (these are only partial schematics, and nothing posted yet for RPi 2, but GPIO header is same as RPi B+)

Numbering below starts from top-row left while holding board with serial port along top edge, so Pin 1 is at top-left corner of board (note this is NOT the same as connector pin numbering on diagrams or board silkscreen, which is an inconvenient arrangement for this purpose)

http://elinux.org/RPi_Serial_Connection

Pin #	Role	Adafruit color
Pin 3	GND	BLACK
Pin 4	host RX	WHITE
Pin 5	host TX	GREEN

Note that these are consecutive, but the first 2 pins are skipped and the RX/TX order is opposite of boards that are compatible with FTDI serial cables, e.g. Minnowboard or BBB

iMX233 OLinuXino Maxi/Mini

Serial port is single-row 4-pin connector U_DEBUG

Schematic: https://github.com/OLIMEX/OLINUXINO/tree/master/HARDWARE

Numbering below starts from the left while holding board with reset and power switches along top edge, so Pin 1 is closest to board edge

Pin #	Role	Adafruit color
Pin 1	host TX	WHITE
Pin 2	host RX	GREEN
Pin 3	GND	BLACK

Note that these are consecutive, but pin order is different from boards that are compatible with FTDI serial cables, e.g. Minnowboard or BBB

Upcoming SELinux Enhancements

2015-07-12T00:00:00-07:00

SELinux userspace policy tools have been undergoing a significant under-the-hood design change in order to make local policy customizations much more practical

Goals of SELinux Userspace Redesign

SELinux is widely used as part of the security architecture of high-security government systems, but its adoption in the commercial space has been hindered by the difficulty that system administrators have in adapting SELinux policies for local needs. This is not entirely surprising, given that SELinux grew out of government projects for improving security, and typical government processes require that specialized security experts be in charge of security configurations, which system administrators maintaining the systems must not change. In contrast, private companies frequently put their system administrators / devops staff in charge of implementing security policies on the servers, and those staff members frequently find SELinux impractical to work with in a more dynamic commercial environment. Some commercial infrastructures are more dynamic than others, but almost all are dynamic compared to the high-security certified-at-every-stage environment from which SELinux was born.

The main gaps in SELinux policy maintenance, hindering commercial usage, include:

lack of mature high-level tools for customizing and reviewing policies
inability to make certain classes of changes with even low-level tools
poor support for preserving local changes across system updates

This current incomplete support for maintaining local SELinux customizations places extra responsibility at the level of policy designer, who should not merely provide secure defaults, but also foresee what parts of the policy might need boolean "knobs" for admins to be able to turn them on and off.

Approach to SELinux Userspace Redesign

The SELinux maintainers have decided to address the gaps in policy maintenance by introducing an intermediate language, quite descriptively named Common Intermediate Language (CIL), between the current text policy definitions and the binary policy format actually loaded the SELinux kernel layer.

Not surprisingly, CIL supports all core constructs existing in the current SELinux policy language, since backward compatibility at a source file level is being provided via a compiler from the old language to CIL. Thus the initial deployment of CIL will not require re-writes to all existing SELinux policies, which should allow a relatively smooth adoption.

Addressing lack of mature high-level policy tools

The current text language used for defining SELinux policy has not proved amenable for the development of high-level tools; the current lack of mature high-level tools is not because high-level tools have never been attempted, but because they never got good enough to gain wide usage and thus become worth maintaining in the long-run. Allowing high-level tools to target an language intentionally designed as an intermediate language is a proven strategy for efficient support of a variety of frontends; see LLVM Intermediate Representation for a shining example of an intermediate language put to good use.

Addressing limitations in low-level tools

CIL introduces advanced namespace support, including inheritance, which enables the development of more powerful administration tools. For an example of the limitations to the current SELinux tools, the existing semanage tool can add a new port label, and subsequently modify or delete its own label, but it can not override a port label created by the installed policy files. Porting semanage to work with the policy at the CIL level will allow it to create custom policy blocks that inherit from the standard policy, but override specific pieces.

Addressing local policy changes and policy updates

With the pre-CIL SELinux userspace implementation, some changes made it necessary to create local policy modules that replaced corresponding default policy modules. There was no automated method of updating those local replacement modules to incorporate changes pushed out as an update to the default policy.

The same CIL features that will allow administrative tools such as semanage to become more capable, providing namespaces with inheritance, will also allow better handling of policy updates by allowing most local changes to be maintained as specific overrides to the default policy. Unrelated changes could then be pushed to the default policy and take effect without conflict.

Early Adopters / Developers

Those adventurous souls who are interested in following progress in SELinux policy management by getting their hands dirty with the latest code from github should consider installing Gentoo for such experiments. The Gentoo SELinux team is already preparing to take advantage of the upcoming changes, and the team lead Sven Vermeulen has provided live ebuilds that allow convenient building of the latest upstream versions of relevant packages. Sven also provides friendly and knowledgeable advice to those working with Gentoo SELinux via the Hardened Gentoo mailing list — the primary avenue of support for new topics such as CIL that are not yet covered by the otherwise excellent Gentoo SELinux documentation.

Run Gentoo or Ubuntu Linux On The New Acer Tegra K1 Chromebook

2015-01-22T00:00:00-08:00

Quick and easy script for making a bootable SDCard loaded with Ubuntu, already stocked with ChromeOS graphics blobs and kernel:

Quick install steps are documented in the readme file here

More info on the chrubuntu script arguments here

Reference for custom kernel build here

Follow the quick install steps in the readme above to make a bootable SDCard with xubuntu.

The Gentoo steps are essentially the same as those documented in the custom kernel reference above, where you can of course build natively on another ARM box, or use the cross-compile method documented above (which is suitably modified for gentoo below).

For Cross-Compiling

First grab the latest crossdev-9999 and the required tools, then build a current armv7a hardfloat toolchain:

# emerge uboot-tools vboot-utils dtc bc -v
# emerge crossdev -v
# crossdev -v -t armv7a-hardfloat-linux-gnueabi

You may need to add the following to your package.keywords file:

sys-devel/crossdev **

Next grab the chromeos kernel sources and configure/build a kernel:

# git clone https://chromium.googlesource.com/chromiumos/third_party/kernel linux-chromeos
# cd linux-chromeos
# git checkout chromeos-3.10

Get the USB firmware (from this overlay) and prepare the config file:

# wget http://commondatastorage.googleapis.com/chromeos-localmirror/distfiles/xhci firmware-2014.11.06.00.00.tbz2
# tar xf xhci-firmware-2014.11.06.00.00.tbz2
# cp -R lib/firmware/ .
# rm -R ./lib/firmware/
# ./chromeos/scripts/prepareconfig chromeos-tegra
# WIFIVERSION=-3.8 ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make menuconfig

According to the reference above, the WIFIVERSION variable is required to build the drivers in drivers/net/wireless-3.8; the mwifiex driver in /drivers/net/wireless/ does not support the Chromebook’s adapter. CROSS_COMPILE should be set to the correct prefix for your toolchain.

In the kernel config, set Device Drivers > Generic Driver Options > External firmware blobs to build into the kernel binary to nvidia/tegra124/xusb.bin. Make any other desired changes, and build the kernel, modules, and dtb file:

# WIFIVERSION=-3.8 ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make zImage
# WIFIVERSION=-3.8 ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make modules
# WIFIVERSION=-3.8 ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make tegra124-nyan-big.dtb

Build a Flattened Image Tree using the provided configuration file:

# wget https://raw.githubusercontent.com/lgeek/gnu-linux-on-acer-chromebook-13/master/nyan-big-fit.cfg
# mkimage -f nyan-big-fit.cfg nyan-big-kernel

Now create the kernel args and a bootable (signed) vboot image:

# echo "noinitrd console=tty1 debug verbose root=/dev/mmcblk1p7 rootwait rw lsm.module_locking=0 net.ifnames=0 rootfstype=ext4" > ./cmdline
# vbutil_kernel --arch arm --pack kernel.bin --keyblock /usr/share/vboot/devkeys/kernel.keyblock --signprivate /usr/share/vboot/devkeys/kernel_data_key.vbprivk --version 1 --config cmdline --vmlinuz nyan-big-kernel

Now insert your prepared SDCard (see below) and copy the signed kernel image to your kernel partition, install the kernel modules, and copy your rootfs; here we’ll use a stage3 armv7 tarball:

# dd if=./kernel.bin of=/dev/sdb6
# mkfs.ext4 /dev/sdb7
# export mount_point="/mnt/card/root/"
# mkdir -p $mount_point
# mount /dev/sdb7 $mount_point
# INSTALL_MOD_PATH=$mount_point WIFIVERSION=-3.8 ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make modules_install
# tar xpf /path/to/downloads/stage3-armv7a_hardfp-20141023.tar.bz2 -C $mount_point

Note

The above partion numbers match the SDCard section below; just use the matching values if yours are different (eg, use 1 and 2 instead of 6 and 7).

Now you can complete the normal Gentoo embedded install, ie, set the root password, configure and enable networking, etc. Insert the card in your Chromebook (make sure it’s in "developer" mode) and Ctrl-U at the warning screen.

For Native Builds

Once you have a bootable SDCard, you’re free to build more components on the card itself, or even better, use an external SSD or hard disk with a USB3 interface. This works well on even the older ARM Chromebooks (velcro, as always, is your friend, along with a short right-angle USB3 cable). For new kernels, simply omit the ARCH and CROSS_COMPILE variables and build/install as above.

Make an SDCard

For either build method, you’ll need an SDCard with the right partition format, which you can do by hand or with the chrubuntu script. The one caveat is that the latest Chromeos on my own Acer 13 is missing some key tools compared to previous revisions, mainly parted and partprobe. The workaround is that you must make a GPT partition table on the SDCard first, using another Linux machine (ie, not your shiny new Chromebook). So first on another Linux machine, insert your (blank or unimportant) SDCard and run the following command, where "target_disk" is most likely either sdX for USB card readers or mmcblkX for built-in devices:

# export target_disk="/dev/sdb"
# parted --script ${target_disk} "mktable gpt"

The following can be done on the chromebook, which is what the chrubuntu script expects. Since we’re not running chrubuntu, we’ll manually create some partitions, one for the kernel image, and one for the rootfs:

# ext_size="`blockdev --getsz ${target_disk}`"
# aroot_size=$((ext_size - 65600 - 33 - 4*1024*1024*1024/512))
# cgpt create ${target_disk}
# cgpt add -i 6 -b 64 -s 32768 -S 1 -P 5 -l KERN-A -t "kernel" ${target_disk}
# cgpt add -i 7 -b 65600 -s $aroot_size -l ROOT-A -t "rootfs" ${target_disk}
# sync
# blockdev --rereadpt ${target_disk}

Note

The above assumes at least a 16 or 32 GB card. 4 GB is known not to work.

Now you can go back to installing Gentoo, or, if you must, run the chrubuntu script and wait…

Stock and Custom PiTFT Kernel Options with Your Choice of rootfs

2014-10-22T00:00:00-07:00

This is primarily useful if you have one of the Adafruit (or similar) small TFT/LCD displays for Raspberrypi (one possible non-Adafruit vendor is SainSmart). The Adafruit displays are available in several sizes and configurations, both touch and non-touch, some come in kit or assembled form, and some are available in both Capacitive and Resistive Touch (CT / RT) varieties. The particular display tested here is the Adafruit 2.8 inch PiTFT resistive touchscreen display. It’s just a bit smaller than the bezel on the Pibow acrylic case, with the button pads exposed along the "bottom" of the board (which needs a default 90 degree rotation argument passed to the kernel).

Since the upstream raspberrypi kernel and firmware does not yet have support for these devices, you’ll need to do one of the following: download a new rootfs or kernel image, update your existing raspbian image using the Adafruit script, or build your own custom kernel using either the modified Adafruit kernel source or adding the fbtft device drivers to your own kernel source and rebuild. The current support is still maintained in the Adafruit forks of the raspberrypi.org repos on Github.

The best documentation I know of for assembling and testing the Adafruit displays are on the Adafruit site, specifically Adafruit PiTFT 2.8 inch Touchscreen Display for Raspberry Pi for the display described here.

Easy Raspbian Install/Upgrade

If you already have a recent Raspbian install, the easiest way is to use the upgrade script provided by Adafruit. Since it’s a Bash shell script, you can read through it fairly easily to see what it will do. Download the script from the Easy Install section and follow the steps in the excellent Adafruit document linked above.

Note

If you don’t have a current Raspbian image, then download the Adafruit image here and burn it to a card (no extra configuration should be required).

If you’d rather do things by hand and/or leave things largely unmodified, then you can manually upgrade your kernel and firmware by cloning the Adafruit firmware repo on Github and manually update your running card as follows, after logging into your pi:

$ cd ~/
$ sudo mount /boot  # make sure /boot is mounted
$ git clone https://github.com/adafruit/rpi-firmware.git
$ sudo cp /boot/kernel.img kernel.bak
$ cd rpi-firmware
$ sudo cp -v boot/* /boot/
$ sudo cp -v extra/Module.symvers /boot/
$ sudo cp -av modules/3.15.8+ /lib/modules/

Depending on your actual hardware, the above kernel config may be missing the correct config to work with the modules and options documented in the Adafruit guide. If the latest Adafruit kernel does not load the fbtft_device module correctly, then download and unpack the .deb packages in the script and manually install the 3.12.x kernel using the above process.

At some point later you can update your userland (vcgraphics) software as well, but for now we only need to test the kernel and display. This may very well suffice for your needs, so feel free to stop here and finish the config manually (or maybe go back and run the install script). The rest of this document is mainly for building your own custom kernel; if that’s what you want to do keep reading…

Custom PiTFT Kernel and Rootfs

Currently the "upstream" for Adafruit hardware support are their repos on Github (of which there are many). In this case, the important ones are their forks of the "official" raspberrypi kernel source and firmware, along with the fbtft framebuffer drivers (which get automatically included as a submodule in the kernel source repo).

https://github.com/adafruit/adafruit-raspberrypi-linux

https://github.com/adafruit/rpi-firmware

https://github.com/adafruit/adafruit-rpi-fbtft

If you’re running Gentoo, you can add the following overlay and use the adafruit-raspberrypi-sources ebuild and ignore the manual clone steps for the kernel, et al, in the next subsection. For Gentoo:

$ cd /usr/local/
$ git clone https://github.com/sarnold/arm.git

Add /usr/local/arm to your PORTDIR_OVERLAY in make.conf, then:

$ echo "=sys-kernel/adafruit-raspberrypi-sources-3.15.9999 **" >> /etc/portage/package.accept_keywords
$ sudo emerge adafruit-raspberrypi-sources

For other distros, continue with the manual steps below.

To build your own kernel and update your boot firmware, you’ll need to clone the first two repos above (if you already have the firmware repo, you can omit the second one below). You’ll also need to either clone recursively or do the git submodule dance to actually bring the fbtft source files into the drivers/video/fbtft directory. Note the following steps assume you’re either building native on the Pi itself, or in an appropriate ARM chroot or virtual environment. If you need to cross-comppile the kerrnel, then you can follow the usual Raspberrypi cross-compile process (you will of course need an appropriate cross toolchain and build environment).

$ sudo -i
# git clone --recursive https://github.com/adafruit/adafruit-raspberrypi-linux.git
# git clone https://github.com/adafruit/rpi-firmware.git
# cd adafruit-raspberrypi-linux
# git checkout rpi-3.15.y -b rpi-3.15-working   # create local branch
# cp arch/arm/configs/adafruit_defconfig .config
# sed -i -e "s/CONFIG_DMA_BCM2708=m/CONFIG_DMA_BCM2708=y/" .config
# sed -i -e "s/CONFIG_DMA_VIRTUAL_CHANNELS=m/CONFIG_DMA_VIRTUAL_CHANNELS=y/" .config
# make oldconfig

There are only two active branches in the adafruit-raspberrypi-linux repo, rpi-3.10.y and rpi-3.15.y so make sure you’ve checked out the branch you want (note 3.10 is default).

Now you can proceed with building the default kernel config, however, you may want to modify it further to suit your own needs (eg, to enable the zram block device or the latest nftables support).

Note

The adafruit-raspberrypi-sources ebuild takes care of the above two sed commands for you, otherwise you must do it manually. This is very important or else the fbtft_device module will not load correctly when following the Adafruit instructions.

Once you’re ready, proceed as you normally would to build/deploy a kernel:

# make menuconfig
# make -jN
# make modules_install
# mount /boot
# cp /boot/kernel.img ~/kernel.bak
# cp arch/arm/boot/zImage /boot/kernel.img
# cp Module.symvers /boot/

If you didn’t follow the above steps to update your boot files, you should do that now (be careful not to clobber your new kernel.img file). Also, with newer u-boot, the above (compressed) kernel zImage file should work fine, but if it doesn’t boot use the (uncompressed) Image file instead (note that you only need one, not both, just rename either one to "kernel.img" when you copy it to /boot).

Now is a good time to reboot and test the new kernel (however, the display won’t really be usable until we finish the configuration). Make sure all your normal services are working and everything looks nominal. If not, copy your backup kernel from above back to /boot/kernel.img and try reverting any suspect config changes (you can always start from the defconfig again if necessary).

As mentioned, we still need to make the required config changes for the framebuffer modules, boot arguments, environment variables, etc. The required changes are all documented in the Adafruit guide above, as welll as in the bash install script for raspbian. You may or may not want all of the changes, depending on how you plan to use your display. For example, if you want to display specific output or other data, simiilar to how you might use one of the 2-line character LED displays, then you probably don’t want to use it as a login console. It all depends on your requirements, so we’ll walk through each major change below.

Although the following configuration snippets are strictly "optional", if you want your display to actually do anything besides light up with a white backlight, then you’ll at least want to configure the module parameters for your display device, and (optionally) autoload the modules on boot.

Configure Modules

Raspbian (and similar) will load the modules specified in /etc/modules, while Gentoo uses /etc/conf.d/modules:

modules="i2c-dev bcm2708-rng snd-bcm2835 spi-bcm2708 i2c-bcm2708 fbtft_device"

The last three are required for the display device, with the following default parameters. Feel free to experiment (within limits) with the clock speed and rotation. Most distros use various conf files in /etc/modprobe.d to configure modules; in this case the Adafruit Software Install section asks you to create /etc/modprobe.d/adafruit.conf with the following contents:

options fbtft_device name=adafruitrt28 rotate=90 frequency=3200000

Note the "name" parameter is specific to the 2.8 inch RT display, while the others are recommended defaults (90 degree rotation puts the button pads along the bottom of the display). These values should take effect whether modules are loaded manually or at boot time.

Configure Kernel Command Line

Modifying the command line will switch the default console device to the small display on boot, so depends on how you want to use the display. With the options below, the display will switch when the fbtft_device module loads (how much of the boot process you can see depends on the size of the kernel log buffer). If you want the console display to switch, then you’ll need to edit /boot/cmdline.txt (which is simply a single line of kernel parameters). Add the last two parameters to whatever is currently there as shown:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait fbcon=map:10 fbcon=font:VGA8x8

Be sure it stays all on one line of text. In Gentoo you can change fonts in your consolefont config, otherwise use "dpkg-reconfigure console-setup" to change your console font.

Configure Xorg

Xorg needs to know both which display to use and what the touchscreen calibration values are. To get started, create a default set of "touch" input values in /etc/X11/xorg.conf.d/99-calibration.conf and use the following values:

Section "InputClass"
Identifier "calibration"
MatchProduct "stmpe-ts"
Option "Calibration" "3800 200 200 3800"
Option "SwapAxes" "1"
EndSection

You can recalibrate later for more accuracy. Somewhere in your environment (either user or system) you also need to set FRAMEBUFFER=/dev/fb1 so X will know where to point the display output. In Raspbian, you can set that in ~/.profile or /etc/profile and will also probably need to move the fbdev-turbo config out of /etc/X11/xorg.conf.d/.

Configure Udev

The actual touchscreen driver for the 2.8 inch RT display will load on its own, however, the following udev rule is recommended. Add the rule file /etc/udev/rules.d/95-stmpe.rules with the following contents:

SUBSYSTEM=="input", ATTRS{name}=="stmpe-ts", ENV{DEVNAME}=="*event*", SYMLINK+="input/touchscreen"

Once you reload the module, the output of "ls -l /dev/input/touchscreen" should be symlink to an eventN device.

Manual Calibration

Follow the Touchscreen Install and Calibration section of the Adafruit documentation to install the tools and manually calibrate your touchscreen (they also have an "auto" calibration script).

Testing Video Playback

Follow the Playing Videos section of the Adafruit documentation to test your display with a small animated video. Since mplayer doesn’t use the accelerated RPi graphics libs, it seems to be right on the edge of being able to play the test video correctly (ie, with audio and video in sync and without dropping too many frames). It seems to play correctly in Raspbian, however, in Gentoo the audio was always several seconds delayed without additional configuration options in ~/.mplayer/config:

lavdopts=lowres=1:fast=1:skiploopfilter=all
vfm=ffmpeg

# ao is the audio output and should be pulse if using pulseaudio
ao=alsa

# af is the audio filter, using re-sampling in this case, to provide audible sound
#af=lavcresample=44100

# Allow framedropping might reduce video quality, but allows video and audio to stay synced
framedrop = yes

fps=24.0

The key option in this case is fps=24 for smooth/synced playback. The CPU utilization is just a little over 50%. Use the above config options with the following command line:

$ mplayer -vo fbdev2:/dev/fb1 -x 240 -y 320 bigbuckbunny320p.mp4

With the above parameters the test video would play with audio and video in sync and decent quality for both audio and video. Note that audio quality can suffer if running from a noisy power source (such as a cheap battery). If audio is noisy, try a decent quality power brick.

Maintaining Your Stock BeagleBone Angstrom Images and Demos

2014-09-14T00:00:00-07:00

In a previous news article, we described building and deploying the latest U-boot and Linux kernels on several armv7 machines, including a BeagleBoneBlack.

In this "HOWTO" we’ll talk about restoring/upgrading/maintaining your stock Angstrom install on both BeagleBone White and Black. Why would we do this? Several reasons, but typically to restore the default install or upgrade to the latest build. The nice thing about BeagleBoneBlack is that Angstrom lives on the eMMC and you can always boot whatever you need off the SDCard. Since BeagleBoneWhite only has an SDCard, you simply swap your default 4 GB card with Angstrom for another (possibly larger) card running your OS/Distro of choice. The two models have slightly different bleeding-edge patch sets and hardware interfaces, but otherwise are very similar:

BeagleBoneBlack Rev A6A - http://eewiki.net/display/linuxonarm/BeagleBone+Black

BeagleBoneWhite Rev A6 - https://eewiki.net/display/linuxonarm/BeagleBone

BeagleBoargd.org info - http://beagleboard.org/static/beaglebone/latest/README.htm

Bonescript and documentation repos - https://github.com/jadonk

Out of the Box

Depending on when it was produced/packaged, your BeagleBone will most likely come with one of the Angstrom builds for SDCard and/or eMMC install. There are some builds which are supposed to support both White and Black, as well as an SDCard image for "flashing" the eMMC on the Black. There are changes in both the Bone101/Cloud9 IDE and Gnome desktop config in the various builds so it’s probably worth it to try a couple of them (some seem to have a less optimal Cloud9 bonescript environment than others).

The recommended builds are given below, "latest" for Black, and a slightly older one for White (recommended for a good new user experience). Simply download the desired image, either from the archive or the latest link, and use the standard tools (dd or WinDiskImager) to uncomnpress and write the image to your SDCard device (be sure and select the .img file unless you want to manually create your card and deploy the rootfs, etc). Use the links above for the manual method, otherwise download one or more of the following images:

Latest eMMC flasher for Black

Recommended SDCard image for White

Archive of OS, u-boot, and kernel images

Latest OS images for all Beagle boards, both Angstrom and Debian

Note

The BeagleBoneWhite Rev A6 I received came with the above 2012.11.22 image on a 4 GB SDCard. The Bone101 slideshow is highly recommended for new users.

Although these kernels are fairly recent, they are several versions behind the latest kernel source (typically 3.10.x rather than 3.16.x) and do not yet support HDMI out. A future HOWTO will discuss building your own OS images with custom kernels, etc.

BeagleBoneWhite Cloud9 IDE and Bone101 Web Tutorial

One of the hardware differences on the White is a built-in USB serial converter on the mini-USB port, so as long as your desktop kernel (or other OS drivers) is configured properly, you should have a serial console as soon as you plug in the USB cable (use minicom or putty to connect at 115200,8N1, no flow control). However, even more interesting is the web server and Cloud9 IDE running on the network interface, which should configure itself if you have DHCP (if not you’ll need to configure it manually via the serial console (login as root with no password).

Download one of the Cloud9-IDE build images and write it to a card:

$ wget http://downloads.angstrom-distribution.org/demo/beaglebone/archive/Angstrom-Cloud9-IDE-GNOME-eglibc-ipk-v2012.05-beaglebone-2012.11.22.img.xz
$ xz -d Angstrom-Cloud9-IDE-GNOME-eglibc-ipk-v2012.05-beaglebone-2012.11.22.img.xz
$ sudo dd if=Angstrom-Cloud9-IDE-GNOME-eglibc-ipk-v2012.05-beaglebone-2012.11.22.img of=/dev/sdX bs=4M

(where sdX is your SDCard device)

Once your BeagleBoneWhite is on the network, use Avahi/Zeroconf or the serial console to discover the IP address, then point your browser at the IP address of your White (anything modern except IE should work). You should see the Bone101 introduction, which will introduce you to, and allow you to interact with, your new BBW. There are bonescript demos and hardware diagrams, including details on the board itself and all the pinouts. Click on the Cloud9 IDE link and try some of the demo scripts (note the BlinkLED demo calls out an external LED, but should also trigger the onboard USR3 LED).

BeagleBoneBlack eMMC Flashing Tool w/ Latest Cloud9 IDE and Documentation

In addition to the bootable micro-SDCard slot, the Black also has onboard eMMC flash (not technically true flash, but consider it like a faster and built-in SDCard). The Black should come out of the box with one of the recent builds listed above; if yours is older or corrupted and you’d like to update to the latest "official" build (including the latest Cloud9 IDE and documentation) then perform the following steps.

Download the latest Cloud9-IDE build image and write it to a card:

$ wget http://downloads.angstrom-distribution.org/demo/beaglebone/archive/BBB-eMMC-flasher-2013.09.07.img.xz
$ xz -d BBB-eMMC-flasher-2013.09.07.img.xz
$ sudo dd if=BBB-eMMC-flasher-2013.09.07.img.xz of=/dev/sdX bs=4M

(where sdX is your SDCard device)

Note

If your BeagleBoneBlack is a Rev C, it should have a 4 GB eMMC, and you can use one of the larger flash images.

Boot the Black with the new card and wait (you can connect a network cable if you like; this will allow you to login and check the status). Once the kernel loads, the LEDs will blink in their default config:

USR0 - blink in a heartbeat pattern

USR1 - light during microSD card accesses

USR2 - light during CPU activity

USR3 - light during eMMC accesses

The USR3 LED will (for the most part) stay lit while the flashing script does the formatting, unpacking, and configuration of the rootfs. When this process is finished, all 4 USR LEDs will light up solid. You can then power down and remove the card, then boot your new Angstrom install.

BeagleBone USB/Ethernet Network Configuration and Automation

Both Black and White BeagleBones default to DHCP for the ethernet interface, so the IP address will depend on your own local network setup. While the White has the serial console on the mini USB interface, the Black has the same port configured as a USB gadget interface. IF your kernel is properly configured, it should appear as 3 devices: an ethernet interface (either usb0 or ethX), an ACM tty device (/dev/ttyACM0), and a USB storage device. Check dmesg or the kernel log for something similar to this:

kernel: usb 2-6: new high-speed USB device number 5 using ehci-pci
kernel: usb 2-6: New USB device found, idVendor=1d6b, idProduct=0104
kernel: usb 2-6: New USB device strings: Mfr=2, Product=3, SerialNumber=4
kernel: usb 2-6: Product: BeagleBoneBlack
kernel: usb 2-6: Manufacturer: Circuitco
kernel: usb 2-6: SerialNumber: 6A-0314BBBK4860
kernel: usb-storage 2-6:1.4: USB Mass Storage device detected
kernel: scsi12 : usb-storage 2-6:1.4
mtp-probe: checking bus 2, device 5: "/sys/devices/pci0000:00/0000:00:13.2/usb2/2-6"
mtp-probe: bus: 2, device: 5 was not an MTP device
kernel: cfg80211: Calling CRDA to update world regulatory domain
kernel: usbcore: registered new interface driver cdc_ether
kernel: rndis_host 2-6:1.0 eth1: register 'rndis_host' at usb-0000:00:13.2-6, RNDIS device, 90:59:af:68:74:cf
kernel: usbcore: registered new interface driver rndis_host
kernel: usbcore: registered new interface driver rndis_wlan
kernel: cdc_acm 2-6:1.2: This device cannot do calls on its own. It is not a modem.
kernel: cdc_acm 2-6:1.2: ttyACM0: USB ACM device
kernel: usbcore: registered new interface driver cdc_acm
kernel: cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
kernel: cfg80211: World regulatory domain updated:
kernel: cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
kernel: cfg80211:   (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm)
kernel: cfg80211:   (2457000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm)
kernel: cfg80211:   (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm)
kernel: cfg80211:   (5170000 KHz - 5250000 KHz @ 80000 KHz), (N/A, 2000 mBm)
kernel: cfg80211:   (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm)
kernel: cfg80211:   (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm)
kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
kernel: nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
kernel: scsi 12:0:0:0: Direct-Access     Linux    File-CD Gadget   0308 PQ: 0 ANSI: 2
kernel: sd 12:0:0:0: [sdb] 144522 512-byte logical blocks: (73.9 MB/70.5 MiB)
kernel: sd 12:0:0:0: [sdb] Write Protect is off
kernel: sd 12:0:0:0: [sdb] Mode Sense: 0f 00 00 00
kernel: sd 12:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

The gadget ethernet device can be configured just like any other, whether manually, with a script, or your distro’s config tools. The latter is recommended; for example on Gentoo you can add a new config file in /etc/conf.d for the ethernet interface (eth1 in this example). You must also allow network hotplugging in rc.conf (or /etc/conf.d/rc if your system is old). The default IP address on the Black side of gadget ethernet is 192.168.7.2, and will assign 192.168.7.1 to the desktop side if left to DHCP (here we hard-code a static address since this is a local subnet).

From /etc/rc.conf:

# This allows all services to be hotplugged
rc_hotplug="*"

And /etc/conf.d/net.eth1:

config_eth1="192.168.7.1 netmask 255.255.255.0 brd 192.168.7.255"
routes_eth1="default via 192.168.7.1"

preup() {
iptables -A POSTROUTING -t nat -j MASQUERADE -s 192.168.7.0/24
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -P FORWARD ACCEPT
return 0
}

postdown() {
echo 0 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -F
return 0
}

It’s also possible to use a udev rule or manual script instead, but I prefer the network-config approach best. The above configuration should configure everything automatically on the host side when the Black USB cable is plugged in. Now you can browse the web interface over the BeagleBone gadget ethernet (just point your browser to 192.168.7.2).

Default BeagleBone USBNet Config Missing DNS Servers and Default Route

If you’re logged in remotely to your BeagleBone over the USB ethernet connection, you might notice there’s only a host route and the default DNS server is the loopback address. If you want your BeagleBone to see outside the local subnet, then you’ll need to add a default route:

# route add default gw 192.168.7.1

You can do this permanently by editing the default dhcp udev rule. You’ll need to edit /etc/udev/rules.d/udhcpd.rules and change this:

SUBSYSTEM=="net",ACTION=="add",KERNEL=="usb0",RUN+="/sbin/ifconfig usb0 192.168.7.2 netmask 255.255.255.252",RUN+="/bin/systemctl start udhcpd.service"

to this:

SUBSYSTEM=="net",ACTION=="add",KERNEL=="usb0",RUN+="/sbin/ifconfig usb0 192.168.7.2 netmask 255.255.255.252",RUN+="/sbin/route add default gw 192.168.7.1",RUN+="/bin/systemctl start udhcpd.service"

and then add your preferred DNS servers to /etc/resolv.conf.

gadgetfs regression workaround

2014-09-01T00:00:00-07:00

The gadgetfs driver has had a module refcount bug in all recent kernels, from v3.10 up through present (last tag from Linus currently being v3.17-rc3). Embedded developers interested in using gadgetfs module with kernels 3.10 through 3.16 may wish to try this version of the patch rather than the one attached to the bug report (the only difference being paths: before or after the gadgetfs driver got shuffled into a "legacy" subdirectory).

liblockdep fix was merged in kernel 3.16

2014-08-30T00:00:00-07:00

The liblockdep userspace lock debugging tool was merged to the mainline kernel tree in version 3.14. Unfortunately, the very next official release, linux 3.15, had a lockdep change that happened to break liblockdep. The patch for this regression was submitted just after Linux tagged 3.16-rc4, and being a fix for a regression (as opposed to a new feature) Linus was still willing to merge it in time for the 3.16 release without any fuss. Users of liblockdep should thus upgrade to 3.16 or later.

Fix for liblockdep regression in kernel 3.15

2014-07-15T00:00:00-07:00

The liblockdep userspace lock debugging tool was merged to the mainline kernel tree in version 3.14. Unfortunately, the very next official release, linux 3.15, had a lockdep change that happened to break liblockdep. I have submitted a patch to fix the regression, but it is currently two steps away from the mainline tree (first Ingo Molnar will need to pull the fix from Sasha Levin, then Linus will need to pull from Ingo). Thus 3.16 might also get released without this fix. Here is a copy of the patch for those who wish to apply it manually to an affected kernel (3.15, and possibly 3.16): liblockdep-fix-regression.patch

Do-It-Yourself Home Automation with the Belkin WeMo

2014-07-13T12:40:00-07:00

By Donald Burr

For many years futurists and sci-fi authors have been promising us the “smart home” that senses and responds to our needs, automating many daily tasks and allowing us to monitor and control every aspect of the house’s operation from anywhere. This so-called “Home Automation” (HA) has been a pipe dream until fairly recently, when the required technology became cheap enough to mass produce. Unfortunately today’s technology still cannot realize some of the more advanced “smart home” concepts — for example, we don’t yet have robots that can perform menial tasks such as cooking, dishes and laundry — but the technology that we do have lets us do some pretty cool — and useful — things.

Commercial HA systems have existed for some time now that can do a wide variety of things. (Some of these are so complex that they pretty closely approach the sci-fi portrayal of the automated home in terms of capabilities, minus robots of course.) However they are proprietary, expensive, and require custom installation, so are out of reach for most hobbyists, newbies and the curious. The good news is that, with HA’s recent growth in popularity (it’s become the topic du jour among the tech press these days) many companies are coming out with some very attractive and affordable HA options. In fact, this sudden growth in HA has not gone unnoticed by the mobile industry, and now major players in the mobile space such as Apple and Google have announced some big HA initiatives. This is a smart move, since the two technologies really go hand in hand — smartphones and tablets serve as perfect control surfaces for interacting with and controlling HA peripherals.

For the longest time, X-10 was the dominant standard in consumer-grade HA, and is still a major player in the industry. A wide variety of X-10 compatible controllers exist, giving users the ability to interface with a large selection of household appliances. However the X-10 standard is quite old (almost 40 years now!) and its communications protocols are slow and inefficient. Today, while X-10 devices are still quite popular, a number of companies have jumped on the HA bandwagon and have released a slew of really cool and innovative HA products. Unfortunately this has created a bit of a “Tower of Babel” situation in that these various devices use different protocols and are (in general) not cross-compatible with each other. Several companies (for example, SmartThings and OpenHAB) have taken the interesting tack of releasing hardware- and/or software-based platforms that “speak” all of the various HA device protocols and therefore essentially act as hubs tying disparate HA devices together — essentially a “Universal Translator” for HA systems.

One of the many companies entering the HA arena is Belkin, the venerable consumer networking giant. Their WeMo devices, which connect to your home network via WiFi, are available in several flavors, including a remotely-controllable power outlet (WeMo Switch,) a remotely-controllable wall light switch (somewhat confusingly named the WeMo Light Switch,) a remotely-controllable power outlet with built-in energy usage meter (WeMo Insight Switch,) a remotely dimmable LED light bulb (WeMo LED,) and, oddly enough, a remotely-controllable Crock-Pot (Crock-Pot Smart Slow Cooker with WeMo) which, as silly as it may sound, actually makes sense since a Crock-Pot is a device you’d want to be able to remotely control and monitor. In addition, Belkin’s webcam has been enhanced with WeMo technology, plus they also make available a separate motion sensor; these devices can be used to trigger other WeMo devices, for example turning the lights on when you get home, or turning on floodlights if motion is detected in your yard. WeMo devices are particularly attractive because they are readily available, reasonably priced and use a simple XML-based REST-like API, and thus can be easily manipulated using Open Source tools.

In this article I will be focusing on the WeMo Switch since that is the one I have; also it is the cheapest of the WeMo devices. I will take you through the procedure for setting up and configuring the device, and will end with a few use cases that should give you an idea of the many things you can do with these devices.

Getting Started

First you will need to buy a WeMo switch. They are available on Amazon which is extremely convenient. Once your WeMo arrives, resist the urge to plug it in right away. Instead, make a note of the device’s MAC address, which is located on a sticker on the back (wall plug end) of the device.

You’ll want to assign your WeMos static IP addresses, since you always want to be able to reach them, and with dynamic IP addresses there is a chance that the WeMo might get assigned a different IP address sometimes. The problem is that WeMo devices don’t support assigning static IP addresses - they always pick up their IP addresses using DHCP. Fortunately, most home routers these days have the capability to assign static IP addresses to a given device based on its MAC address. (Look for a menu option in your router labeled “Static DHCP” or “Reservation DHCP” or something similar.) Then enter your WeMo’s MAC address, and give it a static IP. Once the router has updated its configuration, you can plug in the WeMo.

Next you will need an iOS or Android smartphone or tablet. At this time the only way to perform initial setup on the WeMo (give it a name, tell it how to connect to your WiFi network, etc.) is by using the WeMo app for iOS or Android. (The app is free.) Once the initial setup is complete, you can control the WeMo from any Linux computer. If you don’t have a smartphone or tablet, you can just borrow a friend’s to perform the initial setup. You may want to consider getting one though; as I mentioned earlier, they make excellent HA controllers, plus they have many other uses as well. Decent Android tablets can be had for $150-200 nowadays, and used iOS devices aren’t that much more. (If you are willing to set up a separate unencrypted network for use exclusively by your WeMo devices, there is a way to set up the WeMo without the aid of a smartphone or tablet. This dedicated network need not be connected to the Internet or your home network, but any computers that you want to be able to control your WeMos must be able to connect to it. I would wager that this is something most people won’t want to do, so my advice to you is just borrow a smartphone/tablet or consider getting one.)

Access your mobile device’s WiFi settings screen. You should see a new WiFi access point named "WeMo-*string of random alphanumeric characters*." Connect to it, then launch the WeMo app. It will guide you through the setup procedure, which includes assigning the WeMo a “friendly name” (a name to easily identify this particular WeMo) and connecting it to your WiFi network. Once that’s complete, go back to your mobile device’s WiFi screen and reconnect to your usual WiFi network. If all is well, then when you next run the WeMo app, your new WeMo should appear on screen.

(By the way, as you run the setup procedure, the app will probably inform you that a firmware update is available for your new WeMo. It would be wise to install these whenever they come out, as they are always coming out with bug fixes and improvements. This is another reason why you should have a smartphone/tablet and the WeMo app; it is the only way that you can run firmware updates.)

Now head on over to your Linux PC, and ping the WeMo to make sure it’s alive. Once you have confirmed its aliveness, go off and download, gunzip and install this shell script. (Installation is simple: just copy it to /usr/local/bin or $HOME/bin or wherever you keep random shell scripts.) This script is based on one I found at the Modern Toil blog with some bugfixes and additional features. You’ll also need to have the curl utility installed; this utility is installed by default on most systems, but if not, it should be available in your distribution’s package repository.

Usage is quite simple:

wemo IP_ADDRESS[:PORT] ON/OFF/GETSTATE/GETSIGNALSTRENGTH/GETFRIENDLYNAME

where IP_ADDRESS is the IP address of the WeMo you wish to control, and :PORT (optional) specifies the port to communicate on. (If unspecified, the script will probe for the correct port.) The final argument is the command you wish to give your WeMo. Commands can be entered in uppercase or lowercase. The following commands are available:

ON - turns on the device plugged into the WeMo
OFF - turns off the device plugged into the WeMo
GETSTATE - gets the current state of the WeMo (whether it is turned on or off.) This status is printed to the terminal, and the shell script’s exit code is also set based on this information (0 if off, and 1 if on.)
GETSIGNALSTRENGTH - prints out the signal strength of the WiFi it’s connected to. (Seems to be a percentage, in other words a value of 100 means full signal, 50 means ok signal, etc.)
GETFRIENDLYNAME - prints out the WeMo’s “friendly name.” This name can be set during initial setup and is a handy way of identifying your WeMo devices.

(By the way, for you language junkies, there are WeMo libraries for Perl and Python as well. In fact, I’d wager that there is probably a WeMo library for whatever language floats your boat, I’m sure. Google is your friend.)

There is one final tool you might want to take a look at, a free(!) and extremely useful Web service called IFTTT (“If This Then That.”) This web service allows you to connect many disparate web technologies together using the simple concept of “If a certain trigger is detected/set off, then perform a certain action.” So, for example, you can set up a “recipe” (as they call these formulae) along the lines of, for example, “If you get a new email at your Gmail account, then turn on the flashing police light/siren plugged into your WeMo.” (Hey, nobody said you couldn’t have a little fun with this stuff!) This service is extremely powerful and very useful for non-HA uses as well, so I recommend you check it out.

Now, armed with the above tools, it’s time to check out some use cases. These are ones that I am actually using, and should give you some ideas as to the cool ways you can use these devices.

Use Case #1: The Lights Are On (But Nobody’s Home)

We don’t travel very often, but when we do we are always nervous about leaving the house unattended. In general, thieves pass over houses that look like they have people in them (i.e. there is observable activity, lights turning on and off, etc.) Conversely, they tend to zero in on those houses that always remain dark and look unoccupied. So whenever we go away, we always set up some light timers to turn on and off lights and certain appliances (TV, radio, etc.) at various times throughout the day and evening. The problem with these is that they are ridiculously hard to program. Also, they operate on a fixed schedule - in other words, once you program them to turn on/off at a given time of day, they always turn on/off at that exact time. A determined thief who cases the joint for several days on end could easily notice this pattern and deduce that timers are in use.

With the WeMo, you can easily write up a shell script (or Perl script, or Python script, or whatever) to turn various lights and other devices on or off at a given time. And, since you have the full power of your Linux computer at your disposal, you can do cool things like randomize the times when things turn on or off, or vary the patterns of what devices to turn on or off to further throw off observant thieves (e.g. one day just turn on and off the lights, but the next day throw the TV and the radio into the mix.) Finally, since this “not at home” functionality can be easily disabled when not needed (just stop running your shell script,) you can leave your WeMos plugged in all of the time; there’s no need to go crawling behind furniture to plug/unplug those stupid timers whenever you go away and return home.

Use Case #2: Lazy Man’s Christmas Lights of Doom!

Christmas has always been one of my favorite holidays. Seeing friends and relatives whom I’ve lost touch with over the years, the wonderful foods traditionally associated with the season, and of course there’s the giving (and more to the point, receiving) presents part. But, being a geek, one of my favorite parts of Christmas is the lights. Not surprising, since most geeks I know tend to like things with lots of lights on them, especially blinking ones. Setting up the lights, on the other hand, is not my favorite part of the holiday, and is quite frankly something that I would rather not have to deal with at all. So we foisted this chore off onto someone else. :) (Sorry Scott. Delegation FTW.) Once the holiday was over, we thought about taking down the lights, but we decided that having to go do it all over again when next Christmas rolled around was too repulsive to bear. So we decided to just leave them up year-round, but obviously turned off during non-Christmas months.

Anyway our exterior Christmas lights plug in to an outlet located inside our garage, by way of a super long extension cord. Going out to the garage in the bitter cold to plug them in at dusk and unplug them before going to bed was a real pain. Sometimes, on especially cold nights, I would conveniently “forget” to unplug them before going to bed. I thought about setting up a timer to automate this chore, but those things are a real pain. (See my timer rant above.) Fortunately we ditched our ancient incandescent light string and replaced it with LED Christmas lights a while ago, so if we did accidentally leave them on, it wasn’t a huge energy waste. Still, it is using energy, if only a small amount. Plus it looks kinda weird. Likewise, I was always forgetting to turn off our interior Christmas tree.

Of course, a WeMo would make this task ridiculously easy. This can be accomplished either with a shell script or IFTTT; IFTTT recipes can use the time of day as a trigger, so you can say, “If it’s 5:30 PM, then turn on the Christmas lights.” IFTTT can even determine the time that the sun rises and sets for you if you give it your location (not your exact location, just your ZIP code;) so you can say, “If it’s sunset, then turn on the Christmas lights.”

Use Case #3: New Email Indicator Light (or alarm, or whatever) of Doom!

If you never want to miss an incoming email, it is pretty trivial to set things up so that a WeMo turns on and off when a new email arrives. Perhaps you can flash your desk lamp when an email comes in, or briefly turn on a police siren, or something. If you have the equipment, you can even set up multiple levels of alerts; for example, flash a red lamp if the email is important (comes from your boss or wife) or a white lamp for all other emails. If you have control over your mailserver, then you can easily set up a procmail rule to run the wemo shell script to perform these tasks whenever the desired type of email comes in. If not, then IFTTT can handle that task.

Granted, for most of us this function is perhaps of more value as a novelty than an actual productivity tool. However, for a deaf or hard of hearing individual, this type of visual alert is an absolute necessity.

Use Case #4: Automated “On the Air” Light of Doom!

In my spare time I podcast about Japanese animation, food, travel and culture. A while ago I got myself a cheap “On the Air” sign just like the ones you see in radio and TV studios. I’ll admit it, I did this mostly because it looked cool. But it actually serves as a useful indication to others that I need things quiet, so that people know not to make loud noises or barge into my room unannounced and mess up my recording. Of course, it is something that I needed to manually switch on and off, so I was always either forgetting to turn it on, or forgetting to turn it off once I was done recording.

With the WeMo, I set up a udev rule so that, whenever I plug in my USB microphone, the wemo script runs to turn on the “On the Air” sign; likewise, when I am finished recording and unplug my USB microphone, the wemo script runs again, this time turning off power to the “On the Air” sign. Automation FTW!

Use Case #5: Automatic Cable Modem Resetter of Doom!

Our home Internet connection is generally fast and reliable… except when it’s not. Every now and then, for whatever reason my cable modem decides to go catatonic and suddenly stops communicating with the outside world. Power cycling the cable modem usually brings it back from the dead. This requires me to stop whatever it is I’m doing, wander over to the NOC (my name for the part of my office where all the networking gear is,) and power cycle the cable modem. Annoying, yes, but not overly so. (Honestly, I find the hue and cry of “The Internet is down!!!” from housemates and visitors much more annoying.) But when I’m away from home, this problem turns from being a minor annoyance, to being a critical showstopper. I frequently need to access my home computers while I’m on the road. For one thing, my VPN server, which I use whenever I am traveling and am using a potentially unsafe network, is located on a machine at home. I also often need to log in to and remotely control one of my machines at home, or to fetch a file that I need from my NAS, and so on. Having the cable modem lock up and my Internet connection go down when I’m not at home is a major problem.

I struggled at trying to find a solution to this problem for quite a while. Detecting a net outage was the easy part: I just set up a script that ran periodically from cron that tries to ping a “reliable” Internet server (one that is known to be up most, if not all of the time; Google’s DNS server at 8.8.8.8 is a good candidate.) If the ping fails, then the script assumes that the net connection is down. The problem then became how to remotely power cycle the modem? Since we don’t yet have the full-on HA with robots, a robotic solution was right out. I thought about acquiring an IP power controller. These devices are often installed in remote locations such as radio and TV broadcast towers and allow engineers to monitor power usage and remotely power cycle problematic equipment. However these devices are extremely pricy and way too fancy for my simple needs. Then, one day, I was ranting and raving about my predicament to a friend, and he was like “Dude, just get a WeMo.” (This is, in fact, how I first encountered the WeMo.) With the WeMo in the equation, my script was complete: when it detects a network outage, it commands the WeMo attached to my cable modem to power cycle itself. This setup works like a charm and I have never had an unrecoverable network outage since. Feel free to download and use it yourself. Simply edit the script to suit your particular needs (it is liberally commented) run this script periodically out of cron and you’re all set.

Use Case #6: Automated Pool Reinflater of Doom!

A while ago I picked up a small above-ground swimming pool during a super closeout sale. It’s a popular attraction when our friends with kids come over for a visit, plus it’s nice to float around in when the weather turns uncomfortably warm (which has been happening more often as of late.) This particular type of above-ground pool consists of a vinyl floor and walls, and an inflatable ring at the top of the side walls which helps to hold the walls up and prevents water from spilling over. Unfortunately, this inflatable ring recently developed a slow leak that has proven impossible to locate, even using the old “spray it with soapy water and watch for bubbles” trick. So every other day or so I have to reinflate the ring in order to maintain the pool’s structural integrity, or else risk a flood of Biblical proportions. After a couple weeks of doing this (or, more to the point, forgetting to do this) I cobbled together an automated solution using mostly parts I had lying about. I took the air pump that I already had and attached its hose to the pool ring’s air inlet valve, plugged the pump into a WeMo, and wrote up a quick cron job that turns the pump on for about 10 seconds every other day (just long enough to reinflate the ring to full capacity.) Unfortunately there was nothing in the system to prevent air from leaking back out through the valve when the pump wasn’t powered on. So I devised a one-way air valve (allowing air to flow only into the pool ring) using a PVC check valve combined with some PVC tubing, all put together and sealed using liberal amounts of duct tape. This setup is admittedly a bit ugly, but it works amazingly well, and is in fact airtight. Thanks to this solution, we haven’t found the need to construct an ark in our backyard and gather up as many animals as we could find.

Conclusion

Home Automation is not only useful, it’s just plain fun. And, thanks to a plethora of available devices and falling prices, it is within reach of everybody. Now go forth and automate!

Donald Burr has been using — and abusing — computers and gadgets practically since the day he was born. (Rumor has it he was born with a keyboard in one hand and a mouse in the other.) He is an avid amateur app developer and is available for hire if you need a little help getting started in app writing. When he’s not geeking around with tech, he can be found blogging and podcasting about Japanese animation and culture at Otaku no Podcast. He can be found on the Twitters as @dburr.

liblockdep: userspace lockdep

2014-07-09T00:00:00-07:00

What is lockdep?

Lockdep is the kernel-lock validator. Core kernel developer Ingo Molnar originally designed lockdep to help clean up deadlocks left over from the kernel’s transition in SMP implementation strategy, from a single "Big Kernel Lock" in early days to increasingly fine-grained locking.

Tracking locks necessarily adds some overhead to the code under observation, and thus experienced developers might wonder whether running code with lockdep enabled would tend to change timing enough that race conditions triggering deadlocks would disappear, the "heisenbug" effect.

The good news is that the lockdep validation strategy is not at all dependent on timing — it tracks patterns of lock acquisition and determines whether code could lock if one thread ran code block X while another thread happened to be running code block Y, and generates a warning regardless of whether that concurrency happened or not. Kernel developers can thus use lockdep to track down locking errors without attempting to reproduce an obscure race condition that only pops up on other people’s systems to trigger deadlocks.

Lockdep was officially released in the mainline kernel back in 2006 with version 2.6.18, and it has helped kernel developers clean up many locking bugs since then:

over the past 2-3 years the term "hard lockup" in regression reports has gone down by about an order of magnitude - and much of that can be attributed to the lockdep coverage we have in place -Ingo Molnar

http://lwn.net/Articles/321670/

A very nice overview on lockdep architecture can be found at LWN, and some practical implementation details can be found in the Linux source code as linux/Documentation/lockdep-design.txt

What is userspace lockdep aka liblockdep?

Fast-forwarding from 2.6 kernel days to the 3.13 era, lockdep was still a widely used tool among kernel developers; not only for fixing deadlocks, but also proactively auditing locking sanity of proposed code. Lockdep is lightweight enough to allow fairly normal use of a system while it is enabled, quite a major feature for debugging complicated systems — enough to inspire some jealousy by developers on the other side of the kernel/userspace divide. Free software developers dealing with complicated locking in pthread code needed a lot of patience to run it under the valgrind lock checker.

Finally in early 2013, Sasha Levin decided that since the lockdep algorithm is not really tied to any kernel concepts, the lockdep code could be repurposed as a pthread lock validator. He did this by wrapping the core lockdep code, adding redefinitions and stubs that converted the kernel implementation into a pthread implementation. Ingo Molnar approved of Sasha’s leveraging of lockdep code without uglifying the original kernel implementation, and helped pave the way for the userspace lockdep wrapper (or "liblockdep") to be merged into kernel version 3.14 as part of the tools/ subdirectory alongside other kernel-related userspace tools such as perf.

Unfortunately liblockdep suffered a regression in the very next official release kernel 3.15. I have submitted a patch which Sasha Levin has included in his latest git pull request<pull>_, so hopefully the breakage will be fixed by the upcoming 3.16 kernel release. As liblockdep matures, regressions like this will become unlikely.

For further discussion on liblockdep try this LWN article; there are no official docs in the kernel source code yet.

When is Liblockdep useful

Valgrind helgrind and DRD modules are the main Free Software alternatives to liblockdep for userspace lock debugging, so it is useful to consider what conditions might favor use of liblockdep over valgrind:

consider liblockdep to reduce runtime overhead If the application is complex and the system running it does not have plenty of spare resources (CPU and memory), valgrind may not be practical. For instance, performance is often important when trying to debug an embedded Linux program, either in an emulator (which often adds plenty of overhead itself) or on the native hardware.
consider liblockdep for cross-compile support in niche environments Valgrind has been ported to the most popular architectures and is included in most popular embedded linux build systems (yocto/OE, OpenWrt, buildroot) but that can’t match liblockdep, which should build easily for anything that has a mainlined kernel port.

There are also circumstances that rule out the use of liblockdep:

can’t use liblockdep if the application uses non-pthread locks According to its docs, Valgrind can do lock validation with custom locks, as long as all locking code is suitably annotated.
don’t use liblockdep with recursive mutexes If possible, clean up the code so that recursive mutexes are not needed, but otherwise use something else to debug the locking.

The kernel hates recursive locks. They promote incorrect usage of locking semantics and are completely unsupported in the kernel.

As you might guess from the above, this also means that lockdep doesn’t work at all with recursive locks, and will just detect a trivial deadlock when an attempt to lock a lock twice is made.

Therefore, liblockdep is completely useless in code which relies on recursive locks, and I doubt the situation will change at any point in the future since I don’t see the kernel adopting recursive locking. -Sasha Levin (quoted with permission from an email)

Making Use of Liblockdep

Building liblockdep

If using kernel 3.15 (and perhaps 3.16 which hasn’t been released at the time of writing), start by applying this patch

From toplevel kernel source dir,

patch -p1 < liblockdep-fix-regression.patch

After either applying the patch or determining it is not needed, the build can be run from the toplevel kernel source dir

make -C tools/lib/lockdep

or in new enough releases (> 3.14)

make -C tools liblockdep

For a cross-compile build, just add ARCH and CROSS_COMPILE variables like when building the corresponding kernel, e.g.

make -C tools liblockdep ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi-

or

export ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- make -C tools liblockdep

output files will be found under tools/lib/lockdep/ — liblockdep.a and liblockdep.so.<version> for static and shared libraries respectively.

Testing liblockdep

liblockdep comes with a nice set of unit tests. It is a good idea to make sure the tests pass before trying to use liblockdep to do real work.

Running the tests natively on the build host is quite simple: starting from the toplevel kernel source directory,

cd tools/lib/lockdep
./run_tests.sh

Every test will say "PASSED" if this is a good build of liblockdep.

Running liblockdep

liblockdep build creates both a static library and a shared library, so type of linking can be decided based on needs of the project.

Static library

need to use the static version if application under test is statically linked
watch out for licensing issues — can only ship apps with builtin liblockdep if all the source is GPLv2 compatible (fine to use liblockdep in local test builds with any source though, just can’t release to users if non-GPL)
run application normally, no changes to command line

Shared library

application under test must be dynamically linked to libpthread
use LD_PRELOAD on command line to allow liblockdep to override pthread locking functions
```
LD_PRELOAD=/usr/src/linux/tools/lib/lockdep/liblockdep.so some_thready_application
```

Tips common to both styles of linking

liblockdep traces currently go to stdout, so make sure stdout is not closed
stack traces in output are improved if application is built with -rdynamic (functions will be listed instead of just addresses)
like compiler errors, focus on fixing first complaint from the output, since single error can cascade to generate multiple complaints

Third Early Adopter Release of SailfishOS (1.0.7.16) for Nexus 4

2014-06-30T00:00:00-07:00

As the title and the announcement below says, this is now available for Nexus4 and other devices (see the upstream site for details). Follow the steps below to flash your device and you should see a nice sailing ship on the next boot.

Although it runs on top of an "android" ROM the gesture-based interface is altogether different, and sweet once you get used to it. Try it and see…

Email Announcement

Dear early adopter of SailfishOS for Android devices. This is important - read this whole mail through and follow all steps exactly as written.

IMPORTANT: If you choose to publish this mail through blogs, news sites, forums, or others, quote it as-is and in complete form ONLY, or people’s devices may be at risk.

We’re happy to publish the third Early Adopter Release of SailfishOS (1.0.7.16) for Nexus 4 (mako) to you.

We are still working on the SailfishOS Hardware Adaptation Development Kit, which describes how to port SailfishOS to existing devices based on CyanogenMod 10.1. Newer versions of CM will be supported eventually. We’ll publish the HADK in the very near future.

This installation image is for early adopters only, meaning we know that some things are not functional or perhaps even broken — please see the release notes below. We are excited to get all of you properly included in the early stages of the project. Do note that this SailfishOS image is strictly for personal and non-commercial usage only.

We’ve prepared a ‘demo’ version of the image which contains the kind of preinstalled ‘marketing’ content and the core apps used for demonstrations - this helps you quickly get a feel for all the interactions that are avalable on a device that has been used for a while but isn’t really what you want for personal use. You can however cleanly remove the demo content.

We want to build a community around SailfishOS for Android devices that is based on mutual trust and respect for what we are all doing. Hence — we ask that whenever you do screenshots, videos, forum or blog posts (and we’re happy if you do!) or the like, you emphasise that this is an under-development snapshot and not a final product release.

It is important for Jolla that the correct expectations are set for those who might be users of the final product — and that they understand what they see is not a released product. If you do demo videos, you can take advantage of our new ‘demo content’ image that has pre-set contacts/imagery/messages/etc to show full functionality of SailfishOS.

WARNING: Modifying or replacing your device’s software may void your device’s warranty, lead to data loss, hearing loss, hair loss, financial loss, privacy loss, security breaches, or other damage, and therefore must be done entirely at your own risk. No one affiliated with this project is responsible for your actions but yourself. Good luck.

NOTE: You will lose your on-device data (including /sdcard), so make a proper backup and make sure to copy that backup to your PC.

NOTE: Make sure to read all the release notes below. Please DO NOT contact Jolla Care for any issues encountered with this Early Adopters build, instead use communication channels listed below.

To install this release of SailfishOS on a Nexus 4 device:

Install adb and fastboot

Debian/Ubuntu: apt-get install android-tools-adb android-tools-fastboot

Fedora: yum install android-tools

Mac OS X: Install Homebrew from http://brew.sh/, then: brew install android-platform-tools

Windows: See http://wiki.cyanogenmod.org/w/Doc:_fastboot_intro for instructions

Install Android 4.2.2 (JDQ39) to your Nexus 4

Instructions here: https://developers.google.com/android/nexus/images#instructions

Download links can be found here: https://developers.google.com/android/nexus/images#occamjdq39

Download CyanogenMod 10.1.3 for your Nexus 4

Perform Factory Reset and wipe contents of the /data/ partition in case of leftovers from previous ROMs

The file you want to download is cm-10.1.3-mako.zip

Download links can be found here: http://wiki.cyanogenmod.org/w/Install_CM_for_mako

Download the SailfishOS for Android image for "mako"

The file you want to download is http://releases.sailfishos.org/sfa-ea/sailfishos-mako-release-1.0.7.16-EA3.zip

Another flavour filled with demo content: http://releases.sailfishos.org/sfa-ea/sailfishos-mako-release-1.0.7.16-EA3-demo-content.zip

Install CyanogenMod 10.1.3 on your Nexus 4

Follow the instructions at: http://wiki.cyanogenmod.org/w/Install_CM_for_mako

After flashing the "cm-10.1.3-mako.zip" file, flash the SailfishOS .zip file in the same way ("on top of it")
Reboot bootloader, SailfishOS should boot and be visible

We recommend reading through http://jolla.com/guide/ — some parts may not apply to Nexus 4

If you want to go back to normal CyanogenMod:

Boot into recovery mode
Choose "Wipe data / factory reset"
Flash cm-10.1.3-mako.zip
(to go back to SailfishOS, flash the SailfishOS .zip on top of it)

If you want to go back to stock Android:

Download the stock image from https://developers.google.com/android/nexus/images#occam
Extract the package and follow the instructions for reflashing/re-locking

To SSH into your device via USB (Linux)

Enable remote connection in Settings->System->Developer mode
Set your USB interface on host machine to IP 192.168.2.2
ssh nemo@192.168.2.1
Use the password from developer mode to log in
Use the ‘devel-su’ command with the same password in order to gain root
To SSH over WLAN, use IP listed in developer mode under "WLAN IP address"

Read Sailfish OS release notes: https://together.jolla.com/question/45064/release-notes-software-version-10716-saapunki/

Release notes/Known issues in EA3:

EXPERIMENTAL: Jolla Store is now available, you’ll need to register with your Jolla Account
- NOTE: Booting Nexus 4 with SIM first, and then removing SIM (or vice versa) may cause Jolla Store to see it as two different devices and cause potential breakage. Please stick to either SIM available or not when running SailfishOS on Nexus 4.
  - There may be a bug with oFono RIL support that makes it not report IMEI value causing this and will be sorted out in a later update.
- DISCLAIMER: Using Jolla Store with Jolla Account might break applications on your other devices, use it at your own risk!
- Android support is not available from the Store, even if you can see Android apps listed (those will be removed eventually from store view)
- This functionality means that image comes with only minimal set of pre-installed apps. Use Store to download the ones you need.
The backlight is dark during first launch, but can be fix by switching the currently-not-working ambient light sensor off (uncheck Settings->System->Display->Adjust automatically)
When display is blanked, power management sets WLAN to the lowest speed state
- Can be noticed in a SSH-over-WLAN session
- Chat notifications may arrive with a slight delay

Fixes after EA2:

Watermark removed
Phone-call audio volume can now be changed with the help of volume buttons
Improved responsiveness when waking phone up with the power button
Settings->System->Developer Mode or About Product do not freeze anymore
Reverted to the original (non-Silica) Fingerterm

Fixes after EA1:

Phone-calls with audio work
Timers and alarms (with device powered on) work
HTML5 video+audio works in Browser (tested splash on http://jolla.com )
Update is based on SailfishOS version 1.0.5.16

Release notes/Known issues in EA2:

To securely power off the device, during its boot-up keep Volume Down pressed to enter bootloader mode. Using volume keys, select "Power off" option, then press the Power key
If not auto-detected from SIM, set-up mobile internet data settings via Settings->System->Mobile network->(long tap on the first toggle-item under "Mobile data" section)->(enter settings given by your operator)

Nexus4-specific known issues reported by the adopters (in EA2):

Chinese text input not working
Localhost name is shown as Jolla
Switching between the online and offline status in the status information setting takes very long and often doesn’t switch properly
Google contacts which are put together with different information, are now split up into several contacts in Sailfish
The battery display seems to be a bit buggy because it loses about 15% from one second to another
The calendar overview when filled with events seems to be a bit laggy
The email push is not working correctly, I do not receive any emails until I push the refresh button
Splitting words in the German translation: e.g. in the open apps on the home screen it says: "Kurzzeitmesse" and in the next line the missing "r"
NOTE: all other Sailfish OS issues have already been reported on TJC - http://together.jolla.com - and many of them were fixed in this 1.0.5.16 release

Release notes/Known Issues in EA1:

Developer mode is activated at all times
There has been no throughout testing of telephony related functionality (roaming, airplane mode, etc) and any use of this functionality is at your own risk
Sensors, Device lock, Reset device, Bluetooth, USB control + MTP, Bluetooth, WLAN hotspot, Camera(photography, video recording), and video playback is not supported in this release
The image SW is not currently upgradeable, nor is any typically licensed multimedia codecs (MP3, etc), HERE maps, Android application compatibility layer, or word prediction for virtual keyboard preinstalled
This image does not include any typically licensed multimedia codecs (MP3, etc), HERE maps, Android application compatibility layer, Microsoft Exchange support, or word prediction for virtual keyboard preinstalled
It is not possible to double-tap to wake up the device
Powering off device puts it into a state of deep slumber; possible to get out of by holding power button and volume down key with a bit of persistence
Fingerterm keyboard is not at its best due to the portrait-only mode
FPS drop while scrolling in homescreen due to non-batching when rendering of application icon grid
Icons/graphics appear unproportionally small in browser toolbar, time select widget, and Settings favourite icons
Multiboot/Multirom is not supported yet but we’re happy if you would like to teach/help us

We will all meet in #sailfishos-porters (note, new location) on irc.freenode.net and please use us (thp, alterego, Stskeeps, lbt, sledges) to work together, report any bugs, graphical glitches or missing functionality that you find, which are not included in the release notes above. You can also find the hardware adaptation source code at http://github.com/mer-hybris .

You are also welcome to participate in threads at http://forum.xda-developers.com/nexus-4/general about Nexus 4 and SailfishOS as well as for more general SailfishOS topics at http://forum.xda-developers.com/jolla-sailfish/general

Best regards,
Carsten Munk (Stskeeps) on behalf of the SailfishOS for Everyone team.
Chief Research Engineer @ Jolla

ARMv7 3.15.0 Kernel and U-Boot Images for Gentoo / Other Linux

2014-06-18T00:00:00-07:00

ARMv7 Linux kernel (3.15.0) and U-Boot (2014.07-rc3) images are available for use, built from the github repos of Robert C Nelson, the current upstream maintainer of several board-specifc kernel patch sets, u-boot patches, custom build scripts, and eewiki docs.

The primary (patch) sources used for these builds are:

https://github.com/RobertCNelson/Bootloader-Builder

https://github.com/RobertCNelson/armv7-multiplatform

https://github.com/RobertCNelson/bb-kernel

The primary tested hardware and deployment docs for these builds consisted of:

BeagleBoneBlack Rev A6A - http://eewiki.net/display/linuxonarm/BeagleBone+Black

WandBoard Quad - http://eewiki.net/display/linuxonarm/Wandboard

Udoo Quad -http://eewiki.net/display/linuxonarm/UDOO

The build output and a stage "4" Gentoo rootfs are available here:

http://www.gentoogeek.org/files/armv7-multiplatform/

http://www.gentoogeek.org/files/arm-u-boot-multi/

http://www.gentoogeek.org/files/arm-bb_kernel/

You can always build your own from the above github repos (which is the subject of a future article) and these are really provided just to get started, ie, boot up the hardware, do some test and checkout, maybe chroot using a SATA or USB disk. These choices are left as an exercise for the reader…

Quick Start

For Wandboard quad, download the following, similarly for Wandboard dual, Udoo, etc:

http://www.gentoogeek.org/files/armv7-multiplatform/

3.15.0-rc8-armv7-x1.2.zImage

3.15.0-rc8-armv7-x1.2-dtbs.tar.gz

3.15.0-rc8-armv7-x1.2-firmware.tar.gz

3.15.0-rc8-armv7-x1.2-modules.tar.gz

Download the latest u-boot image for your specific board from the appropriate subdir:

http://www.gentoogeek.org/files/armv-u-boot-multi/

u-boot-wandboard_quad-v2014.07-rc3-r8.imx

For BeagleBoneBlack, use the above BBB u-boot image, but substitute the following kernel:

http://www.gentoogeek.org/files/arm-bb_kernel/

3.15.0-bone1.zImage

3.15.0-bone1-dtbs.tar.gz

3.15.0-bone1-firmware.tar.gz

3.15.0-bone1-modules.tar.gz

Follow the eewiki doc (above) for the appropriate board to make an sdcard, and choose a rootfs. For a basic Gentoo stage "4" rootfs with a portage tree and a few other tools, download the following tarball and edit the password file, hostname, and net config. Otherwise try one of the distros recommended on the eewiki doc.

Gentoo armv7a-hardfloat stage 4:

http://www.gentoogeek.org/files/armv7-multiplatform/

stage4-armv7a_hardfp-rootfs-20140614.tar.bz2

Extra Details

Download the kernel and u-boot

Download all of the kernel and u-boot files for your board; each board has a separate directory for u-boot images (choose the version you want) whereas the armv7-multiplatform kernels support multiple boards (again, choose the version you want). If you have a BeagleBoneBlack, you can try the multiplatform kernel, but the bb_kernel repository patches seem to be more current.

Download a rootfs

You can choose a Gentoo Stage 3 tarball (or the stage 4 listed above) or any other appropriate rootfs (ie, Debian Wheezy armhf). Substitute whatever rootfs tarball you like when you get to that step. The stage 4 provided here is essentially the latest Gentoo stage 3 for ARMv7 hardfloat, with some extra apps (distcc, ccache, gkrellm, nfs-utils, screen, ntp, zram-init, portage tree).

See the eewiki links for supported distros, or the Gentoo mirror list for ARM stage 3 tarballs and portage snapshots. Download the stage 4 at the link above.

Prepare an SDCard

Follow the steps for your board/device; the key difference lies mainly in which u-boot files are used and how they are installed. It isn’t strictly required to reformat the card each time, depending on what your needs are, ie, you can preserve an existing rootfs and manually update the bootloader and/or kernel as needed.

If the card seems slow when running, then check top and look for a piggy process (sometimes the mmc driver can be seen taking up too much cpu).. Some cards can feel faster or slower depending on how they’re formatted, so feel free to test some of the recommended formatting options (although the partition allignment should be fine with default fdisk; whether or not you want ext4 to use a journal and setting a specific stride option is up to personal choice).

Lastly, make sure your mount options include relatime/diratime or similar for decent card performance. Also consider using a small zram swap device and/or tmpfs device for /tmp if you have enough physical RAM. The latter options actually help quite a lot for general performance and on most compile jobs, but make sure your parallel make setting isn’t too high when you compile things like binutils/gcc/glibc, and especially webkit-gtk and firefox…

The Mercury Programming Language

2014-05-24T00:00:00-07:00

High-level discussion of the Mercury programming language, covering its origin and how it is categorized in various common language-classification schemes

Infrequently Asked Question (IAQ): What is the Mercury programming language, and what is the point of it?

The Mercury programming language was developed in Australia, by the computer science department at the University of Melbourne. It was funded by various grants. The code for the reference implementation, developed there, is all delivered under the GPL license, i.e., entirely open-source. There are other implementations. The Mercury project has left the University of Melbourne, and now may be accessed at the website www.mercurylang.org.

Programming languages and their features are classified by computer science in a number of categories. The ideas summarized here are well covered by Wikipedia and other net sources, not to mention possibly an education in computer science.

Low-level versus high-level

A low-level language requires closer attention to the specific hardware features of the computer system running the program than a higher-level language would demand. At the lowest level is the binary command set for the cpu. C and C++ are higher-level than that, and Fortran, Ruby and Ada are high level languages. Although many higher-level languages have special features allowing contact to specific hardware features, it is easy in these languages to write programs that run properly with little or no attention to the specific hardware it runs on. Mercury is a high-level language.

Practical versus research

A practical computer language is directed toward usage by those who are interested in getting data processed at a reasonable time and cost. They are interested in answers, not arcane computer science questions. They want their analysts and programmers to write software with a minimal amount of bugs and which is not hard to maintain. A research language supports scientific investigations, either computer science or any other investigations requiring something special not easily available using the usual languages (like Fortran, C, Visual Basic, Perl, Python etc.). In order to satisfy unique research requirements, investigators will put up with a remarkable amount of inconvenience with respect to obtaining an exotic language, building the compiler or other support software, struggling with the documentation, and so on. This is not what the commercial user wants to deal with. Mercury aims to be a practical language.

Compiled versus interpreted

This refers to the expected run-time environment for the language. The same program could be either compiled or interpreted, depending on what run-time software is used. When a language is compiled, it is transformed to a form which can be run as process under the operating environment of the computer system. When a language is interpreted, it is transformed to data which is to be used by an additional program, the interpreter, which is running on the target computer system. The introduction of the interpreter as a middleman results in additional consumption of computer resources (time and/or space) when the program is run. Mercury is a compiled language. This is in the interest of speed, as Mercury aims to be used for large practical problems.

Imperative versus declarative

Computer languages started out all being imperative. That means the main emphasis is on the algorithms used to process the data. There has been a movement among both computer scientists and actual software developers in which some persons want the computer languages they use to more closely reflect the problems they are solving. This is in opposition to closely reflecting the computer architecture they are using, and constitutes a shift toward a higher level. Some speak of "compilable requirements", meaning that by the time you have an accurate requirements specification there is enough exact information available that a computer could produce the acutual runnable program. Lisp is a good example of a declarative language, as is Prolog, which is a logic language. Mercury is in the declarative camp.

Iteration versus recursion

Iteration refers to repeating a process on some data by repeating an action on each chunk of data. In some languages, iteration is not available directly. Instead, operation on a big structure is accomplished by finding that the big structure is made up of a bunch of structures to the big structure, so that the routine after acting with respect to the big structure calls itself to deal with the similar component structures. (If this seems confusing, it is not surprising—most people not already familiar with recursive methods need more than an introductory paragraph to get comfortable with it). Mercury uses recursion, not iteration. This goes with being a declarative logic language.

Object-oriented programming

This is a technique in which the problem environment is modeled in terms of "objects", which are entities each of which have data attributes and operations upon them. Some languages do not directly support this concept: examples, C or Fortran. Some languages provide support for this style of programming, but it is not exactly built into the language, and it is not necessary to progam in an "object-oriented" way. Examples: Perl, ocaml. Some languages are defined from the ground up for object-oriented programming. Examples: C++,Python,Ruby,Java. Mercury is like Perl or ocaml, in that features are provided to aid programmers who wish to adopt an object-oriented style.

Functional programming

This refers to languages which organize computations into functions which have the property that the result of applying the function algorithm to its inputs depends only on the inputs to the function. This means that a routine cannot have so-called "side effects", that is, do something not implied just by the inputs. Printing out results is a side effect, and so are various other things you might want your computer to do. The implication is that you can have the bulk of your code purely functional, but you must also somewhere have a part that is not purely functional. For some languages this involves a few contortions, but this is OK if they are minimal, and you gain the desired benefit of the rest of your code (presumably the hard part) being especially easy to understand and maintain. Mercury is a functional programming language like ML, Haskell or ocaml. It has very precise tracking of the necessary "impure" code.

Logic programming

Logic programming languages address the problem of finding valid consequences of given axioms and postulates. These languages are based on mathematical logic, comparable to, for instance relational database being based on set theory. Logic programs have application in artificial intelligence. They also have been used to prove that certain programs are valid, i.e., are doing what they are supposed to. An example would be showing an encryption program is valid. Program proving is so difficult it is not widely done. "Research Continues" Mercury is a logic programming language.

Backtracking versus other

Backtracking refers to the feature of a language which applies when at a certain point the computation can proceed along alternative paths (each of which may require extensive calculations) but it is not known which (if any) will provide an acceptable result. The backtracking feature allows a checkpoint to be taken so that if this path does not work out, the computation revert to before the alternatives, and another alternative can be tried. This could be all arranged by the programmer, but it is immensely easier when the language supports it. Mercury provides backtracking, like Prolog or transaction rollback in SQL.

Compile-time type-checking versus run-time type-checking

In programs, data structures are assigned types, which determine what operations may be performed on them. It is easy to make mistakes, and write a programming command which specifies an invalid operation. There are three options for what happens next: 1. The error is not found except possibly by detection of bad output when the program is used. This is the worst outcome. 2. The error is found when the program is run, because the program contains checks which signal that an invalid operation was attempted, or that the result of an operation was invalid. Here at least we know that there was a problem. 3. The error is found when the program is compiled. The programmer can fix the problem before delivering a bug. Which option is desirable depends on how important is is for the program to be reliable. Option 2 means a lot of work by the programmer if reliability is to be achieved. However, it is not hard to get programs to compile, and compilers are relatively cheap to develop and thus to buy. Option 3 requires plenty of work to develop a compiler, and demands careful type definition and use by the programmer. All this is to detect bugs earlier and make it less likely that a bug will be delivered. Mercury, like Ada or ocaml, performs extensive compile-time checking.

Mercury is the result of adding functional programming (like Haskell) and full compile-time type checking (like Ada or ocaml) to logic programming (like Prolog). Some might look at this as having the both the desirablity and the likelyhood of a successful effort to hybridize armadillos and hyenas. Nonetheless, the project has produced a running compiler and a full build environment. It works on Linux and Windows and some other places. The compiler is fast and so are the resulting programs (relatively speaking).

Yet Another Git Intro: A basic introduction with command-line examples

2014-05-03T00:00:00-07:00

Getting Started

Installing git

Fedora 7, Ubuntu 8, and later: The git-core package is available through the standard package repositories. Other Linux and BSD environments should be similar, eg, emerge git on Gentoo will install the needed commands and dependencies. If your platform does not package git, you can download the latest stable release from "http://www.kernel.org/pub/software/scm/git/". Windows users are recommended to install either TortiseGit or Cygwin (the latter includes a bash shell and many other Linux programs).

Basic Tasks

First, note that you can get documentation for a command such as git log —graph with:

$ man git-log

or:

$ git help log

With the latter command, you can use the manual viewer of your choice; see the git-help(1) man page for more information.

You should configure git with your name and preferred email address before doing any operation. The easiest way to do so is:

$ git config --global user.name "Your Name Here"
$ git config --global user.email you@domain.example.com

To download a new package from a git repository:

$ git clone https://username@github.com/VCTLabs/vct-web.git

To avoid the username part of the above URL, use a .netrc file to store your login ID, or use an ssh public key on github and the ssh URL:

$ git clone git@github.com:VCTLabs/vct-web.git

To update a package to the latest upstream version ("fast-forward merge"):

$ git pull origin <branch-name>

or more simply, to pull from the default branch/location from which you cloned:

$ git pull

will pull from the origin repository and default branch defined in the package-name/.git/config file.

One way to undo all local modifications:

$ git checkout -f

To check in your own local modifications (e.g. update the web site, do some refactoring, fix a bug, or apply a patch):

$ vi vct-custom.css pelicanconf.py

Note

Don’t forget to run ‘git add’ and ‘git rm’ if adding or removing files.

To check in all local modifications to your local repository:

$ git commit -a -m "added new stylesheet and updated pelican config for enabling plugins"

Undo recent commits

If the commits are local only, then "git reset" is probably appropriate. OTOH, if the commits are already public, ie, they’ve been pushed to a remote repository and potentially cloned by someone else, then "git reset" is most likely not the right answer. That said, if you’re working completely by yourself, then any method is viable (again, depending on what your goals are).

To make one or more commits go away cleanly when working with others, the right tool is almost certainly "git revert". You can specify one commit or a range, and git will make a new commit that exactly reverts the changes made by the specified commit(s). Suppose you wanted to get rid of two commits, and you’ve already made two new commits (on top of the bad ones) that you want to keep. First, get the commit hashes for the two bad commits, then revert them:

$ git log --oneline | head -n 4

498e425 added three new drafts, still need metadata
87b2a14 latest updates to static pages
6dcdf91 added artile template with example rst metadata
6fcd2c0 Add note for ubuntu users to use apt-get version of pip instead

$ git revert 6fcd2c0..6dcdf91

Sometimes you have made a few commits, or just pulled a change, and simply want those commits to go away completely:

$ git reset --hard HEAD~2   # erase last 2 commits

This will essentially erase the top two commits, as if you had never made them. DO NOT do this, if you’ve already pushed said commits (at least not without coordination with others who may have pulled those commis). Note that this is quite different from git revert, which applies a reversed patch as an additional commit.

Listing changes in your working dir, in diff format

Display changes since last ‘git add’ or ‘git rm’:

$ git diff

Display changes since last commit:

$ git diff HEAD

Obtain summary of all changes in working dir:

$ git status

List all commits on the current branch, with descriptions:

$ git log

The ‘git log’ option "-p" shows diffs in addition to commit messages. The option "—stat" shows the diffstat.

List all commits to a specific file:

$ git log content/pages/contact.rst

Branches

Basics

List all local branches (add -a to see remote branches too):

$ git branch

Make desired branch current in working directory:

$ git checkout $branch

Create a new branch from master, and make it current:

$ git checkout -b alternate-theme master

Examine which branch is current:

$ git status

(‘git branch’ also shows you the current branch, using a "*" in front)

Obtain a diff between current branch, and master branch

In most trees with branches, .git/refs/heads/master contains the current ‘vanilla’ upstream tree, for easy diffing and merging. (in trees without branches, ‘master’ simply contains your latest changes). The following is equivalent to git diff HEAD, when used with HEAD branch:

$ git diff master..HEAD

Obtain a list of changes between current branch, and master branch:

$ git log master..HEAD

(this is equivalent to git log, when used with HEAD)

Rather than full changeset descriptions, obtain a one-line summary of each changes:

$ git shortlog master..HEAD

Merging changes from one branch to another

Suppose that you do work on two different branches, and after work on those two branches is complete, you merge the work into master:

$ git checkout master       # switch to branch master
$ git merge drafts          # merge drafts into master
$ git merge new-theme       # merge new-theme into master

Misc. Topics

Optimize your repository

git is heavily optimized for fast storage and retrieval on a per-command basis. However, over a long period of time, it can be useful to perform further optimizations, including packing all git objects into single "packfile" for fast retrieval and less wasted disk space. The following:

$ git gc

will optimize your repository. You don’t need to run this frequently — git is quite fast even without it. See the ‘git gc’ man page for more details.

Don’t forget to download tags from time to time

Doing a "git pull" only downloads new commits from the remote, and updates the requested remote head. This misses updates to the .git/refs/tags/ and .git/refs/heads/ directories. For tags, run git fetch —tags in your local repo.

Tagging a particular commit

In many cases, you will want to give interesting or significant commits a name, known as a tag. The Linux kernel uses tags for each kernel version: "v2.6.21", "v2.6.22", etc. For example, to create a new tag after a particular commit:

$ git tag my-tag

This creates a new tag named "my-tag", based on the current commit. You can also make an "annotated" tag, or a GPG-signed tag, so read the man page for more details.

Mobile App Development

2014-04-25T23:40:00-07:00

So You Wanna Write an App?

Jumping feet first into the wild world of app development

by Donald Burr

It happens to all of us eventually. Perhaps you’re having a frustrating time using an app, and wish that the developer could have written it this way, or added that feature. Maybe a brilliant idea for the Next Big Thing(TM) hit you while you were in the shower one day, but you are hesitant to hire a programmer to help realize your dream, lest they decide to run off and steal your idea (or perhaps you just can’t afford to hire anybody.) Or maybe you’re trying to get something done, but as it turns out, there just isn’t “an app for that.” (It happens more often than you think!)

In cases like this, you might have wished that you could write your own app. But the mere thought of doing so sends a chill of fear and dread up your spine. It’s hard! I have to learn how to code! It costs a lot of money! What if Apple rejects my app?! Well, fear not my friend: I am here to tell you that developing apps for iOS and the Mac is not as scary as it sounds, nor is it as costly as you might think. In fact, most of the tools and information you’ll need are entirely free of charge!

“Whoa there! Wait a minute!” you might be saying to yourself right about now. “I thought it costs money to get into the Mac and iOS developer programs!” This is in fact correct: you must pay Apple $99 per year to join the Mac and iOS developer programs. (That’s $99 per program folks; so if you want to develop for both Mac and iOS, you’re talking $198/year.) However, you only need to pay if you wish to do the following:

sell your apps in the App Store
sell your Mac apps outside of the App Store with a registered developer ID
test your iOS apps on actual iOS devices. (Apple’s developer tools come with a software-based iPhone and iPad emulator; if all you want to do is use this, then you need not pay anything.)

In other words, if you’re just getting started, or just want to experiment (get your feet wet, so to speak,) you don’t have to pay anything at all. Only once you’ve decided that, yes, this developing apps thing is my kind of gig, do you have to pay up.

But paying up does have its perks. For one thing, you get to test new versions of iOS and Mac OS long before the public sees them. This is of course vital to ensure that your apps will work with the “next big thing”; but it’s also fun. You also get access to the full Apple Developer website, with its wealth of information, technical documents, and training videos. (all session videos from Apple’s Worldwide Developers Conference (WWDC) are available for your perusal; it’s the next best thing to being there!) You also get access to the Developer Forums, an extremely useful place where you can chat with other developers, ask questions, and discuss the finer points of app development. (Oftentimes you’ll find Apple engineers monitoring the forums, so you can get your help from the experts!)

So What Do You Need?

First and foremost, you will need a Mac. Odds are that, if you are reading this, you probably already have one. If not, then go and get yourself one. Fortunately it doesn’t have to be a particularly powerful Mac; however it does need to run the current operating system of the day (which at the time of this writing is OS X 10.8 Mountain Lion.) A Mac mini will do nicely, as will a MacBook Air. Even a used Mac would work brilliantly, so long as it can run Mountain Lion.

You will also need Xcode, Apple’s integrated development environment. This comprehensive set of tools comes with everything you need to write, test, debug and package your app for sale or distribution. Xcode is available for free in the Mac App Store.

Finally, while not strictly speaking a requirement, a cadre of friends, acquaintances, coworkers, etc. who would be willing to help out by testing your apps on their precious iDevices is extremely helpful. The more testing your app gets, and the more variations on hardware and software that it is tested with, the better.

Dipping Your Toe in the Shallow End

Psst! Did you know that you can create iOS apps without even writing a single line of code? It’s true! Using the RedFoundry web-based environment, you can quickly and easily create a basic iOS app by dragging and dropping “widgets.” You can choose from a wide variety of widgets, such as RSS feed viewers, audio players, Twitter stream viewers, Flickr viewers, and more. As you build your app, you can quickly test it on your actual iDevice using Red Foundry’s free app. When your app is all done, you can publish it to the App Store straight from the Red Foundry website.

Red Foundry lets you easily build an iOS app by dragging and dropping widgets.

There are many similar tools out there; for example GameSalad is specially geared toward writing games, and includes such features as physics engines, graphics sprites, collision detection, and so on. Services such as RedFoundry and GameSalad are a great place to start, and are perfect if all you want to do is write a fairly simple app.

The beginnings of the Next Big Game. I’m calling it “Belligerent Avians.”

In GameSalad, you can easily assign attributes and set up behaviors for your game’s objects.

Time to Swim Out to the Deep End

Although tools such as Red Foundry and GameSalad are quite capable, there will come a time when what you want to accomplish is beyond their capabilities. It’s time to leave the shallow end and swim out to the deep end of the pool, and learn how to code. But, as I said earlier, it really isn’t as hard as it sounds.

A good place to start is the excellent book Objective-C Programming: The Big Nerd Ranch Guide, written by the fine folks over at the Big Nerd Ranch. Big Nerd Ranch is a company that, among other things, teaches seminars on Mac and iOS programming; they also write apps on a contract basis. So they know their stuff. This book assumes you know nothing about programming at all, and teaches you the fundamentals of programming in Objective-C, the language Mac and iOS apps are written in. It also teaches you the basics of Mac and iOS programming. After this, you might want to check out their other books for more in-depth coverage of the particular platform you are interested in developing for: Cocoa Programming for Mac OS X: The Big Nerd Ranch Guide (for Mac development) and iOS Programming: The Big Nerd Ranch Guide (for iOS development.)

No, you’re not looking at the Matrix. This is Xcode’s code editor. It has many features that help make entering code a lot easier.

Your local community college or Adult-Ed center might have courses on computer programming and app development that might help as well. With the growing popularity of iOS devices and Macs, many institutions of higher education are jumping on the bandwagon and offering such courses.

Using Xcode’s Interface Builder, you can construct your app’s User Interface by dragging and dropping from a palette of buttons, text fields and other common controls.

Or, if you are interested in iOS development, you might just want to stay at home and study online. One of the best courses on iOS app development, CS193P is available for free on Apple’s iTunes U service (one of the most under-appreciated parts of iTunes in my opinion.) This college-level course is taught at Stanford by instructors who really know their stuff; and since Stanford is practically in Apple HQ’s backyard, you can often find Apple employees appearing as special guest lecturers. Using the free iTunes U app you can watch the lecture videos, view the lecture notes and slides, take notes and view the homework assignments. (You can also download all the code you see the instructor working on in class, but you’ll need to do that from your computer.) It’s all the benefit of a college education without the hassles (being graded, quizzes, finals, having to get up at the crack of dawn to make it to class on time, etc.)

iTunes U: All the benefit of a college education, without the hassle.

Tips for Success (and Avoiding App Rejection)

You’ve probably heard the horror stories of developers toiliing long and hard on their masterpieces only to get flatly rejected by Apple. It’s true that Apple does reject a fair number of apps; however for each app rejection, there are also thousands of apps that “made the cut.” You only hear about the rejections because they complain about it vociferously on Twitter and blogs. (In other words, they are a very vocal minority.)

But you would certainly want your app to be as good as it could be, and to really shine when viewed by the Apple review personnel; and to help with that I have the following suggestions.

Beauty is more than skin deep. Apple products are known for their beauty, elegance, simplicity and ease of use. The same applies to the apps that run on them. Apps should pick a design aesthetic and adhere to it. User interfaces should be as simple as possible and convey only the most necessary information. Navigation should be intuitive and fluid. Every year at WWDC, Apple hands out its Apple Design Awards to apps that they feel exemplify the design and usability aesthetics of Apple. You would do wisely to study these apps and learn from their success. There is also a corollary to this tip, which is:
If you want to create the best apps, use Apple’s tools. Tools such as Red Foundry, GameSalad, etc. make pretty decent looking user interfaces; but when compared to apps written using Apple’s tools, they look a bit “off.” The user interface is “kinda, sorta, almost” Mac- or iOS-like. But it’s just different enough that someone would notice. For the most “Mac-like” or “iOS-like” experience, you really need to roll up your sleeves and dive into the Apple development environment.
Keep your ear to the ground… Wayne Gretzky once said, “A good hockey player plays where the puck is. A great hockey player plays where the puck is going to be.” Great apps come not from responding to the desires of users today (although that is certainly a good thing to do too,) but from anticipating what a user’s needs will be in the future. Keep an eye out on the various social networks. Find out what the “next big thing” is ? the next big social network phenomenon that is starting to gain traction; trends in the types of games people like to play; etc. When the “Next Big Thing” hits, you’ll be ready with “an app for that.”
…but be prepared to jump out of the way if a train comes rushing by. Apple has been known to radically (and without much warning) change things in its software or its developer policies. Don’t be surprised if a feature or programming trick that you are relying on suddenly changes its function or stops working entirely. Be ready with a workaround. Joining the paid developer program really helps in this regard, since you get early access to new iOS and Mac OS releases before the public does; you can test your apps, change your code or create workarounds to counter Apple’s changes, and be ready with an app update when the public gets their next OS update.
Play by the rules. Heed Apple’s developer agreements. Most importantly, only use public APIs (the programmatic interfaces that Apple vends out.) A good rule of thumb is: if it’s documented in official Apple documentation, then it’s fair game. If, on the other hand, you heard of this really cool trick from a buddy, or saw an interesting function call while digging through Apple’s code, then I would stay very far away. Using private or undocumented APIs or coding tricks is the quickest way to get your app rejected. And Apple, given its fickle nature, has been known to change their developer rules suddenly and without much notice (if any at all); so stay alert!
Respect your users’ privacy and security. Apple has made some tremendous strides in respecting a user’s privacy, and so should you. If you don’t need a particular type of information (a user’s location, access to his/her contacts or calendars, etc.,) then don’t ask for it. If you do, provide a good reason ? Apple gives developers a way to inform the user as to why your app is requesting that information from him. If you transfer any sensitive data online, encrypt it. If you need to store any sensitive data on a user’s phone (usernames and passwords, etc.) do so securely. Apple’s Keychain is an excellent and very secure method of safely storing usernames and passwords, and there are many code libraries (both from Apple as well as third parties or Open Source) suitable for use in encrypting data for transmission over the Internet. And if a user decides not to trust your app with access to his/her information, please respect their decision.
Apple gives you a lot of toys to play with. Use them. Apple gives us developers a bevy of toys to play with, and users will expect you to take advantage of them. Got an app that deals with significant amounts of data (databases, note taking apps, etc.)? Use iCloud; many people have multiple iDevices nowadays, and expect their data to be available ubiquitously. Are you writing a game? Then by all means use GameCenter! Not only is it a great way to give users value (people love challenging each other and boasting about their high scores,) but it is also a great way to promote your game. Are you writing an app with regularly updating content (blog, magazine, newspaper, etc.)? Then you should definitely be in Newsstand. And so on. And finally:
Don’t be disappointed if you don’t make millions overnight. (OK, this doesn’t really have any bearing on your app’s chance at approval. I just didn’t have anywhere better to put this.) The fact of the matter is that for every Angry Birds and Tweetbot, there are thousands of unsung heroes ? otherwise good (perhaps even great) apps that just didn’t manage to grab peoples’ attention and interest. It’s all a numbers game, and with over 700,000 apps currently in the app store, getting noticed is pretty difficult. Sadly the App Store is not as fertile a ground for making money as it was in the past. That’s not to say that you don’t have any shot at all; it’s just going to take some effort (and some good luck) on your part. Talk about your app as much as you can on Twitter and Facebook without being annoying; show it off to friends, relatives, colleagues, guys off the street, anyone really; perhaps even call in to a certain nationally syndicated radio talk show host. But if you go into this whole app-making thing with the sole desire to strike it rich, then you have lost the game. Rather, the attitude I like to take is similar to the attitude I take toward podcasting: “Do it because it’s fun and/or interesting. Do it as a pasttime or a hobby. And if you happen to make some money, more power to ya.”

A Note about Accessibility

Finally, let me talk a little about accessibility. (Besides, a certain podcaster and “technology for the disabled” advocate would have me drawn and quartered if I didn’t!)

There are 314 million people worldwide who have some form of visual disability (partially sighted or totally blind) 278 million have moderate to severe hearing loss. That’s a lot of users ? more users than are on Facebook! ? and shutting yourself out of that sizable user base is sheer insanity. Especially when adding accessibility to your app is a lot easier to do than you think (In fact, in most cases Apple has already done most of the work for you!) The iPhone is the world’s most accessible smartphone. Period. Same goes for the Mac. Apple has put a ton of effort into making their platforms accessible, and they are definitely the industry leader. These devices have literally opened up a whole new world for the disabled, and you would be foolish not to enable them to use your app.

That’s it! No, really, that’s it!

Don’t get me wrong. There will certainly be a fair amount of head-scratching and “WTF” moments as you dive in and learn the intricacies of coding. But it’s certainly not an insurmountable challenge, and definitely isn’t something that you should run away from in fear. Aside from the resources I have mentioned already, there are a ton of places where you can get help and more information online. (Besides the Apple Developer Forums, Stack Overflow is an excellent place where you can ask programming-related questions and get really good answers.) Programming is sort of like learning to ride a bicycle: you’ll probably be quite a bit wobbly at first, you’ll fall a lot and suffer many scrapes and bruises; but pretty soon you’ll get the hang of it and will be zooming through code like nobody’s business.

Now go forth and realize your dreams! Who knows? Maybe you’ll end up writing the next Angry Birds… er, I mean Belligerent Avians!

Dos and Don’ts for profiling and performance optimization

2014-04-17T00:00:00-07:00

Tips

Note

Do profile your code to look for performance bottlenecks, but don’t worry about hand-optimizing your code right away. Readable and obvious is much better from a maintenance standpoint than obtuse or cryptic. Also, if you put too much into one line of code, you may actually be defeating the code coverage analysis.
Do think about Big "O" when making algorithm selections, but don’t worry about trying to out-think the compiler optimizations. Trust in gcc/g++ at least up to the -O2 optimization level, but don’t use -O3 (which is known to break certain constructs).

Using gprof

Compiling for profiling

Before you can profile your program, you must first recompile it specifically for profiling. To do so, add the -pg argument to the compiler’s command line:

$ g++ -g -pg -o dummy dummy.cc

This creates an executable called dummy from the source file dummy.cc with debugging and profiling turned on. Another way to do this is to add -pg to the CFLAGS line in your Makefile.

Creating gmon.out

Once your program has been compiled with profiling turned on, running the program to completion causes a file named gmon.out to be created in the current directory. gprof works by analyzing data collected during the execution of your program after your program has finished running. gmon.out holds this data in a gprof-readable format.

Things to keep in mind:

If gmon.out already exists, it will be overwritten.
The program must exit normally. Pressing control-c or killing the process is not a normal exit.
Since you are trying to analyze your program in a real-world situation, you should run the program exactly the same way as you normally would (same inputs, command line arguments, etc.).

Running gprof

Run gprof like this:

$ gprof program-name [ data-file ] [ > output-file ]

If you don’t specify the name of a data file, gmon.out is assumed. Following the gprof command with "> output-file" causes the output of gprof to be saved to output-file so you can examine it later.

For this example, the program name is dummy and we will save the output into a file called dummy.output:

$ gprof dummy > dummy.output

Analyzing gprof’s output

After completing the last step, the gprof’s analysis has been saved into the dummy.output file. You can use your favorite text editor to examine this file. By default, two kinds of analysis are performed: the flat profile and the call graph. Both types are explained in the following sections.

Interpreting the flat profile

The flat profile shows the total amount of time your program spent executing each function. At the end of the profile, you will see a legend describing what each of the columns of numbers means. Here is some of the output from the flat profile:

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls  us/call  us/call  name
 37.50      0.15     0.15    48000     3.12     3.12  Life::neighbor_count(int, int)
 17.50      0.22     0.07                             _IO_do_write
 10.00      0.26     0.04                             __overflow
  7.50      0.29     0.03                             _IO_file_overflow
  7.50      0.32     0.03                             _IO_putc
  5.00      0.34     0.02       12  1666.67 14166.67  Life::update(void)
  5.00      0.36     0.02                             stdiobuf::overflow(int)
  5.00      0.38     0.02                             stdiobuf::sys_write(char const *, int)
  2.50      0.39     0.01                             ostream::operator<<(char)
  2.50      0.40     0.01                             internal_mcount
  0.00      0.40     0.00       12     0.00     0.00  Life::print(void)
  0.00      0.40     0.00       12     0.00     0.00  to_continue(void)
  0.00      0.40     0.00        1     0.00     0.00  Life::initialize(void)
  0.00      0.40     0.00        1     0.00     0.00  instructions(void)
  0.00      0.40     0.00        1     0.00 170000.00  main

Note that the functions mcount and profil (profil does not appear in this listing) are part of the profiling aparatus; their time gives a measure of the amount of overhead due to profiling. Also note that functions like stdiobuf::sys_write and _IO_do_write are part of the system libraries and not directly part of your code.

In this output, we can see that 37.5% of dummy’s execution time is spent in Life::neighbor_count. This is the highest percentage for any function in the program. It is also worthwhile to note that it gets called 48,000 times. This is your first hint that Life::neighbor_count might be the biggest bottleneck in the code.

Interpreting the call graph

The call graph shows how much time was spent in each function and its children. From this information, you can find functions that, while they themselves may not have used much time, called other functions that did use unusual amounts of time. Like for the flat profile, a legend appears after the call graph describing what each of the columns of numbers means.

Here is some of the output from the call graph:

                         Call graph (explanation follows)

granularity: each sample hit covers 4 byte(s) for 2.50% of 0.40 seconds

index % time    self  children    called     name
                0.02    0.15      12/12      main [2]
[1]     42.5    0.02    0.15      12         Life::update(void) [1]
                0.15    0.00   48000/48000   Life::neighbor_count(int, int) [4]
---------------------------------------------------------------------------
                0.00    0.17       1/1       _start [3]
[2]     42.5    0.00    0.17       1         main [2]
                0.02    0.15      12/12      Life::update(void) [1]
                0.00    0.00      12/12      Life::print(void) [13]
                0.00    0.00      12/12      to_continue(void) [14]
                0.00    0.00       1/1       instructions(void) [16]
                0.00    0.00       1/1       Life::initialize(void) [15]
---------------------------------------------------------------------------

[3]     42.5    0.00    0.17                 _start [3]
                0.00    0.17       1/1       main [2]
---------------------------------------------------------------------------
                0.15    0.00   48000/48000   Life::update(void) [1]
[4]     37.5    0.15    0.00   48000         Life::neighbor_count(int, int) [4]
---------------------------------------------------------------------------

The lines full of dashes divide this table into entries, one for each function. Each entry has one or more lines.

In each entry, the primary line is the one that starts with an index number in square brackets. The end of this line says which function the entry is for.

The preceding lines in the entry describe the callers of this function and the following lines describe its subroutines (also called children when we speak of the call graph). If the caller of a function cannot be determined, <spontaneous> is printed instead.

The entries are sorted by time spent in the function and its subroutines.

In this example, we see that the first entry is for Life::update, the second entry is for main, and so on. 42.5% of the program’s execution time is spent in Life::update and its children. Life::update only has one child, Life::neighbor_count. In the fourth entry, we see that Life::neighbor_count consumes 37.5% of the program’s execution time and has no children. As in the flat profile, the call graph shows that Life::neighbor_count was called 48,000 times.

Based on this information and what we observed in the flat profile, we can conclude that Life::neighbor_count is the main bottleneck in dummy.

For more detailed information on gprof, check out the gprof Manual.

Dos and Don’ts for generating code/test coverage data (for C, C++, Java, and Python)

2014-04-03T00:00:00-07:00

Tips

Note

Do generate coverage data for your code; it should be especially helpful for pointing out where test drivers are exercising code (and where they aren’t). Don’t rely on just line or branch counts and shoot for higher numbers. For exmple, use the branch count information to make sure you’re executing all the appropriate decision paths, etc.
Do try it under "normal" or ad-hoc execution conditions to look at the main flow(s) through your code. Don’t restrict your use of coverage data to testing only.

The current tools described here include the following:

gcov/lcov (for C, C++, and pretty much anything compiled with gcc)
coverage.py (for Python)
cobertura (for Java)

Using gcov

Once your code is at least running local unit tests, and if you see any performance issues, then you can add the profiling targets to your makefile or python code. For non-kernel code, you should add a make target by following this simple example:

$ gcc -fprofile-arcs -ftest-coverage tmp.c
$ a.out
$ gcov tmp.c
90.00% of 10 source lines executed in file tmp.c
Creating tmp.c.gcov.

Please try some of the available gcov options, such as:

-a —all-blocks

Write individual execution counts for every basic block. Normally gcov outputs execution counts only for the main blocks of a line. With this option you can determine if blocks within a single line are not being executed.

-b —branch-probabilities

Write branch frequencies to the output file, and write branch summary info to the standard output. This option allows you to see how often each branch in your program was taken. Unconditional branches will not be shown, unless the -u option is given.

-c —branch-counts

Write branch frequencies as the number of branches taken, rather than the percentage of branches taken.

-f —function-summaries

Output summaries for each function in addition to the file level summary.

Generating coverage data for the kernel

To generate coverage data for the Linux kernel, it’s as easy as rebuilding the kernel with the following options enabled:

CONFIG_GCOV_KERNEL=y
CONFIG_GCOV_PROFILE_ALL=y

and mounting the debugfs file system under /sys/kernel like so:

# mount -t debugfs none /sys/kernel/debug

Then change to the kernel source tree:

# cd /tmp/linux

and run the coverge tool on one or more source files:

# gcov kernel/gcov/base.c -o /sys/kernel/debug/gcov/tmp/linux/kernel/gcov/
File 'kernel/gcov/base.c'
Lines executed:52.17% of 46
kernel/gcov/base.c:creating 'base.c.gcov'

Code coverage information for the specified source file(s) can be found in the files created by gcov. Alternatively, use LCOV to obtain the information automatically.

Quick start using coverage.py

Install coverage.py from the coverage page on the Python Package Index, or by using “easy_install coverage”. For a few more details, see Installation.

Use coverage run to run your program and gather data:

$ coverage run my_program.py arg1 arg2
blah blah ...your program's output... blah blah

Use coverage report to report on the results:

$ coverage report -m
Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
my_program                   20      4    80%   33-35, 39
my_other_module              56      6    89%   17-23
-------------------------------------------------------
TOTAL                        76     10    87%

For a nicer presentation, use coverage html to get annotated HTML listings detailing missed lines:

$ coverage html

Then visit htmlcov/index.html in your browser, to see a nicely formatted report.

Tracing Cheatsheet

2011-11-26T00:00:00-08:00

List of simple methods to enable tracing in various scripting languages (shell, python, ruby, perl, tcl) and build tools (make, ant, cmake, scons). Basic tracing is normally possible without editing the code or installing additional debugging software.

Script Tracing

Tracing shell scripts

Trace every line, showing literally when read and after shell expansion whenever executed:

$ export PS4='+(${BASH_SOURCE[0]}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
$ bash -v -x <script>

Tracing goes to stderr, so can be saved to a file separate from normal program output on stdout, e.g. replace second line with

$ bash -v -x <script> 1>/tmp/OUT 2>/tmp/TRACE

Within the trace output, lines prefixed with ‘+’ are post-shell-expansion and the others are pre-shell-expansion.

In order to trace all the way down into scripts called from a toplevel script, the toplevel script will need to be edited to run them with ‘-v -x’ options.

NOTE:	When using zsh rather than bash, you should be able to use the same commands except for replacing ‘${BASH_SOURCE[0]’ with ‘${(%):-%N}’ in the ‘export’ command.

Tracing python scripts

Trace everything:

$ python -m trace --trace <script>

Trace except for standard library modules:

$ python -m trace --trace --ignore-dir=/usr/lib/python<py-version> <script>

where <py-version> is python version from ‘python —version’ minus the trailing patch level, e.g. 2.7 or 3.2 rather than 2.7.4 or 3.2.2

Trace except for specific modules:

$ python -m trace --trace --ignore-module=<module 1> --ignore-module=<module 2> <script>

Tracing ruby scripts

Trace every line:

ruby -rtracer <script>

Tracing perl scripts

Trace every line:

$ export PERLDB_OPTS="NonStop AutoTrace"
$ perl -d <script>

Trace subroutine entry and exit points, displaying arg list:

$ export PERLDB_OPTS="NonStop frame=14"
$ perl -d <script>

Same, but copy trace to a file (can’t just redirect stdout/stderr):

$ export PERLDB_OPTS="NonStop frame=14 LineInfo=TRACE.out"
$ perl -d <script>

Tracing tcl scripts

Trace everything:

$ tclsh
% <paste excerpt below>
% source <script>

using this as the paste contents:

rename proc _proc
_proc proc {name arglist body} {
    uplevel 1 [list _proc $name $arglist $body]
    uplevel 1 [list trace add execution $name enterstep [list ::proc_start $name]]
}
_proc proc_start {name command op} {
    puts "$name >> $command"
}

Recipe taken from "Trace" section of the debug page of the Tcler’s Wiki, which is a page worth exploring further if needing to do much tcl debugging.

Build Tracing

Tracing make builds

Normal tracing:

$ make --debug -n <target>

Note that ‘-n’ is a simulation mode so commands are not actually run, but it shows _all_ commands that would have been run (even commands that would otherwise be quieted by ‘@’ prefix).

Much improved tracing:

$ make SHELL='$(warning [$@ ($^) ($?)])bash'

This is essentially a suggestion from an excellent Dr. Dobbs article on Make debugging with the additional observation that often it can be passed in on the command line without editing the Makefile (unless some or all of the Makefiles are overriding SHELL already).

Tracing ant builds

Normal tracing:

$ ant -v <target>

Tracing cmake builds

Normal tracing:

$ cmake --trace <target>

Tracing with extra debug info:

$ cmake --trace --debug-output <target>

Tracing scons builds

Normal tracing:

$ scons --taskmastertrace=<logfile> <target>

where logfile can be a ‘-‘ to print dependency trace to stdout

tcpdump cheatsheet: command-line capture of network traffic

2011-05-18T00:00:00-07:00

tcpdump cheatsheet for command-line capture of network packets, useful for cases where where using wireshark for capture is not practical (e.g. network troubleshooting on remote servers or embedded devices)

tcpdump cheatsheet

Modes

tcpdump has two main modes

text display on console (but can be saved to file with redirection)
binary capture to file for detailed analysis (stored in pcap format)

Overall syntax

tcpdump <options> <filter expression>

where both options and filter expression are optional

Using just the plain command,

tcpdump

tells tcpdump to use default settings

Default settings

listen on first network interface (which is often "eth0")
text display on console (as opposed to saving to file)
display summary of info from each packet header
display all packets
unlimited capture (don’t quit until killed, e.g. with ctrl-c)

Common options

which interface: -i eth0, -i eth1, -i usb0, -i any ("any" is a virtual interface that includes packets from all interfaces)
skip name lookups for ip addresses or port numbers: -n (use this if DNS is having problems so that name lookups are slow or failing)
limit number of packets to capture: -c 100, -c 2000
display packet data in addition to header: -X
limit data size for each packet: -s 100, -s 2000 PROBABLY DO NOT WANT THIS WHEN STORING PACKETS FOR LATER ANALYSIS!
write captured packets to file: -w filename
read from existing file (instead of live capture): -r filename
verbose: -v, -vv, or -vvv for increasing verbosity (when capturing to file, any of the above usefully enables a count of received packets)

NOTE:: when writing packets to file, full contents of packets are saved in pcap format; flags such as -X are only needed if displaying on console, when the default display summarizes header info and skips packet payloads altogether

Filter syntax

limit by ip address/hostname: host x.x.x.x, host somehost.com
limit by port number/port name: port 80, port http official port names can be found in /etc/services
limit by packet type: ip, tcp, udp, icmp, arp, esp, ah (last couple being IPSEC packets)
combine simple limits into more complex criteria: not, and, or
group operators in complex criteria: ( ) good idea to use quotes around the whole filter statement if it is complicated enough to need parentheses…

Examples

display summary of packets to/from eth1 network card, skipping hostname lookups

tcpdump -i eth1 -n

display summary of packets to/from any web server

tcpdump -i any port http

save summary of ping packets to/from google.com to a file

tcpdump -i any icmp and host google.com > /tmp/google_ping.txt

capture all traffic to a file for later analysis (e.g. in wireshark)

tcpdump -i any -w /tmp/saved_traffic.pcap -v

capture all traffic except your own ssh connection to a file

tcpdump -i any -w /tmp/saved_traffic.pcap -v "not ( port ssh and host me.org )"

Posix-style Regular Expressions

2009-09-24T00:00:00-07:00

Posix regular expressions cheatsheet, where "posix" refers to the syntax supported by traditional unix utilities (grep, sed, awk) without the additions of perl-style "extended" syntax

Regular expression syntax

Pattern	Backslash needed (basic syntax)	Meaning
.	No	ANY character
[:class:]	No	Character inside pre-defined set named "class" [1]
[abc]	No	Character inside given set
[a-m]	No	Character inside given range
[^abc]	No	Character NOT inside given set
(abc)	Yes	Cluster into pattern group
a\|b	Yes	Alternative patterns/groups
a?	Yes	Repeat count: occurs zero or once
a*	No	Repeat count: occurs ANY number of times (including 0)
a+	Yes	Repeat count: occurs one or more times
a{2,5}	Yes	Repeat count: occurs some number within range
^abc	No	Anchor: beginning of line
abc$	No	Anchor: end of line

[1]	there is no character class named "class", merely serving as placeholder for the real classes of which the most important are: alpha, digit, alnum, lower, upper, blank, and space

Basic syntax vs extended syntax

The only difference between "basic syntax" and "extended syntax" [2] is differing policies on backslash-ification.

Basic syntax requires certain characters to be backslashed to obtain the special pattern-matching meaning ("Yes" in middle column of table above), while other characters are assumed to have the special meaning unless they are backslashed to be meant literally ("No" in middle column of table above).

Extended syntax is much more consistent: omit the backslash to use the pattern-matching meaning for all special characters, or include a backslash to take the literal meaning of a character.

[2]	"extended" in this case refers to Posix-style extended syntax, not to perl-style extended syntax, which adds entirely new syntax elements not included in this cheatsheet

Expression grouping

Reasons for grouping with parenthesis:

Apply count to whole group (abc)?
Use groups as alternative choices (abc)|(xyz)
When using regexes to do substitutions, groups can be recalled as part of replacement text

Tips

Convenient testing from the command-line

Use —colour option of egrep or grep to do testing with a candidate pattern

# matches "2014-02-15"
echo "2014-02-15: did stuff" | egrep --colour "^[^:]+"

Develop complicated patterns in extended syntax with egrep, then backslash-ify to basic syntax and test with grep if needed

# matches "product ID 4234A"
echo "quantity 5: product ID 4234A-Z99" | egrep --colour "(product ID [0-9]+)+A"

# matches "product ID 4234A"
echo "quantity 5: product ID 4234A-Z99" | grep --colour "\(product ID [0-9]\+\)\+A"

Convenient testing from a browser

A very friendly web app for interactive experiments with regular expressions: small text box for the regex, large text box for the text to be matched, and highlighting that shows all matches in real-time

http://regexr.com/

Watch out for "*" vs "+"

Choose carefully between "any number" and "one or more"; it is a common mistake to use "any" when the match should specify "at least one"

# matches "A", but probably not intended
echo "HUMMA" | egrep --colour "(product ID [0-9]*)*A"

# no match
echo "HUMMA" | egrep --colour "(product ID [0-9]+)+A"

Relevant man pages

"man 7 regex", "man perlre"

Command-line Editing with Readline

2007-12-28T00:00:00-08:00

Cheatsheet for readline command-line editing, aimed towards bash but most functions are generally applicable to other shells and apps that use readline

Readline usage

Display configuration

Use bash ‘bind’ builtin command:

bind -v  # show readline options
bind -p  # show readline functions and corresponding key mappings
bind -q yank  # show key combination for function yank

Details:

bind -v shows current settings in native readline command format; each line is a valid readline command for setting an option, e.g. if the option ‘bell-stype’ currently has value ‘visible’, it will be displayed like this:

set bell-style visible

bind -p shows functions and key mappings in native readline command format; each line is a valid readline command for mapping a function to a key combination, e.g. if ‘ctrl-y’ key combination is bound to the function ‘yank’, it will be displayed like this:

"\C-b": history-search-backward

bind -q <function> will show all the key combinations that will trigger the given function (if any), e.g. if the function ‘yank’ is bound to ‘ctrl-y’ key combination, it will be displayed like this:

yank can be invoked via "\C-y"

Modify configuration

Use bash ‘bind’ builtin command:

bind 'set bell-style visible'             # select visual bell-style
bind ' "\C-b": history-search-backward'   # map history-search-backward function to ctrl-b
bind -r "\C-k"                            # ctrl-k is unmapped

Details:

when setting an option or mapping a function to a key combination, the syntax matches the output of the corresponding display command, but with a pair of single quotes surrounding the whole readline command (so that it will be interpreted by bash as a single entity)
when mapping a key combination, don’t allow whitespace before ‘:’ or mapping may silently fail
bind -r <key-combo> can be used to remove a default function mapping from a key sequence that tends be accidentally triggered by a habitual typo

To make changes to settings or mappings persistent, either put commands as above in ~/.bashrc or edit ~/.inputrc (creating it if non-existent). The syntax in .inputrc is the same as the arguments to ‘bind’ command, except in the case of unmapping a key combination (the ‘bind -r’ syntax being just a shortcut):

# select visual bell-style
set bell-style visible

# map history-search-backward function to ctrl-b
"\C-b" : history-search-backward

# ctrl-k is unmapped
"\C-k" : self-insert

The advantage to using .inputrc instead of .bashrc is that custom keybindings should be re-usable in other applications that use readline, rather than just bash

Reusing history

previous-history (up arrow, ctrl-p)
next-history (down arrow, ctrl-n)
history-search-backward (unbound by default)
history-search-forward (unbound by default)
reverse-search-history (ctrl-r)
forward-search-history (ctrl-s -- poorly chosen default)
yank-nth-arg (@<number> alt-ctrl-y)
shell-expand-line (alt-ctrl-e)
magic-space (unbound by default)

Details:

previous-history
- goes backwards through previous commands in historical order
- one of the most commonly used readline functions, being conveniently mapped to down arrow by default
next-history
- goes forwards through previous commands in historical order
- only useful for reversing direction after going backwards in history via another function (sorry, no time-traveling into future here)
- one of the most commonly used readline functions, being conveniently mapped to up arrow by default
history-search-backward
- searches backwards through commands starting with the sequence of characters already typed
- rarely used, due to being unbound by default
history-search-forward
- searches forwards through commands starting with the sequence of characters already typed
- only useful for reversing direction after searching backwards in history via ‘history-search-backword’ (sorry, still no time-traveling)
- rarely used, due to being unbound by default
reverse-search-history
- searches backwards through commands including the sequence of characters typed after calling function
- can alternate between calling reverse-search-history or forward-search-history and typing more letters to narrow down the search
forward-search-history
- searches backwards through commands including the sequence of characters typed after calling function
- can alternate between calling reverse-search-history or forward-search-history and typing more letters to narrow down the search
- only useful for reversing direction after going backwards in history via another function (nope, no time-traveling)
- has a positively unusable default binding of ctrl-s, which is interpreted by terminals as XOFF flow-control signal and stops all input until crtl-q = XON is typed

yank-nth-arg
- without number arg, yanks 1st word from prev command
- with number arg, yanks that word from prev command (numbering starts with 0)
shell-expand-line
- expands current command-line using bash expansion rules, allowing further editing
magic-space
- performs history expansion and inserts space after it

Completions

complete (tab)
menu-complete (unbound by default)
dynamic-complete-history (alt-tab)

Details:

complete
- attempts to complete current word, if enough has been typed to be unique
- one of the most commonly used readline functions, being conveniently mapped to tab by default
menu-complete
- rotates through possible completions on successive presses
dynamic-complete-history
- uses history words as possible completions for current word

Editing commands

transpose-chars (ctrl-t)
undo (ctrl-_)
revert-line (alt-r)

Details:

transpose-chars
- swaps current character with previous
undo
- undoes last change to line
revert-line
- undoes all changes to line
- useful when going back to a line from history and accidentally mangling it
- not as useful when starting from a fresh line (unless you just want to clear it…)

Movement Commands

beginning-of-line (ctrl-a)
end-of-line (ctrl-e)
backward-word (alt-b)
forward-word (alt-f)

Readline on results of history expansion

set bash histverify option

shopt -s histverify

when using history substitution (e.g. "!!" and so forth), hit ‘return’ and see the command after the substitution, edit if desired using normal readline features, and hit ‘return’ again to finally accept the command

Overview of the GNU autotools approach to source code configuration

2006-03-30T00:00:00-08:00

Overview of GNU autotools from two different perspectives: software builder and software developer (specifically, one maintaining a project’s build system)

GNU Autotools from two perspectives

GNU autotools, or commonly referred to as plain "autotools", provides source code configuration, allowing code to be customized in certain ways as part of the build process. GNU autotools build-time customization is an important part of the toolset for many Free Software projects, who use it to

implement compile-time optional features
account for differences in build environments

Let’s look at autotools from two different perspectives, the "software builder" and the "software developer".

The software builder might be an end-user who is daring enough to try out some software that has not yet been packaged up by the distro of their choice. Or, it might be someone on the packaging-team of a distro, working to turn the latest source code release into convenient packages for end-users. It could even be a developer for another project, who needs a really fresh version of some dependency used by their own code.

The software developer perspective in this case covers anyone on the software project team who helps maintains the build process, in particular one that allows source code to be built across a wide variety of hosts. This goal of allowing source code to be built on hundreds or thousands of different hosts across the world rather than just needing to build on a small number of tightly-controlled corporate build servers has been a strong factor behind the spread of autotools among Free Software projects rather than simpler, but less-flexible, solutions such as hand-written Makefiles.

Software Builder’s view of autotools

Autotools is rather friendly and convenient from the software builder’s perspective, with a basic invocation that has been memorized by many a power-user of Linux:

./configure ; make ; make install

and additional customization power available when it is needed:

./configure —enable-ssl —prefix=/opt/newstuff ; make ; make install

The configure script:

accepts arguments to allow user to customize software
runs tests to characterize build environment and locate build dependencies

assuming all tests were successful,

creates Makefiles that are adapted to
- available build tools and locations of libraries
- customization choices made by the user
(optional) creates config.h which has platform-dependent information that allows source code to adapt itself to
- properties of the host platform
- customization choices made by the user

Software developer’s view of autotools

Autotools from this perspective is significantly more complex than what the software builder typically experiences. A source code release supporting the convenience of "./configure ; make ; make install" typically has

a configure script that
- offers appropriate options for customizing package
- locates any external build dependencies (utilities, libraries, headers)
- runs tests to detect platform characteristics
- uses collected information to turn Makefile templates into actual Makefiles
- (optional) uses collected information create config.h from a template
corresponding Makefile templates
(optional) corresponding config.h template

The autotools suite is a framework for generating each of these pieces, so each new project does not have write code from scratch to handle each aspect of customization (command-line parsing, system characterization, and applying the resulting configuration to build files and source code)

Input files maintained by developer

toplevel
- configure.in OR (deprecated) configure.ac
  - m4 macro calls with shell script fragments as needed
  - lists user-configurable options, system characterization tests, and files to be generated from templates by variable substitution
- (optional) acinclude.m4
  - custom m4 macro definitions
  - can define system tests not distributed with autotools
per directory
- Makefile.am
  - file lists and variable definitions
  - use @NAME@ placeholders for variables to be substituted by configure

Autotools utilities

autoscan:	generate a preliminary configure.in by scanning source code for common portability issues
aclocal:	collect all m4 macros needed by configure.in into aclocal.m4
autoconf:	create configure script based on configure.in (with required macros in aclocal.m4)
automake:	create Makefile.in files based on configure.in (with required macros in aclocal.m4) and the corresponding Makefile.am
autoheader:	(optional) create config.h.in from which configure script can generate config.h
libtoolize:	install helper files used by libtool to abstract away platform-dependent details of building libraries
autoreconf:	calls the various autotools in appropriate order

Relationship between tools and files

Prepare for configuration

aclocal(configure.in + acinclude.m4 + autotool files) = aclocal.m4
autoconf(aclocal.m4 + configure.in) = configure
automake(aclocal.m4 + configure.in + Makefile.am) = Makefile.in
autoheader(configure.in) = config.h.in
libtoolize [installs some libtool files into toplevel dir]

Perform configuration

configure(config.h.in) = config.h
configure(Makefile.in) = Makefile

Perform build

make(Makefile + libtool files + source files) = build products

Resources

Autotools basics:
	http://sourceware.org/autobook/autobook/autobook_toc.html
Autotools Mythbuster:
	https://www.flameeyes.eu/autotools-mythbuster/
Autoconf macros archive:
	https://www.gnu.org/software/autoconf-archive/

Yet another vim cheatsheet: vim basics

2005-02-24T00:00:00-08:00

vim cheatsheet that covers editing basics (and which mostly applies to other vi implementations as well)

vim basics cheatsheet

This cheatsheet is intended to help those who "barely can get around" in vi to become competent vim users. It is not intended for complete beginners, since an understanding of vi editing modes is assumed.

Most of these commands come from original vi, and theoretically will work in simple vi applications rather than just vim. "Theoretically" refers to a caveat that many embedded systems run an extremely cut-down busybox vi, leaving out a lot of supposedly standard vi features (sometimes explicitly warning that a feature is "not implemented", and sometimes just doing a weird edit instead of the intended one).

Notable vim-specific commands included in this list for being incredibly useful when available:

visual selection commands, in particular, column-selection mode for convenient editing of text in fixed-width columns
macro recording and playback, which can simplify repetitive editing tasks so easily that it has replaced many a throw-away perl/python/ruby/whatever script

MOVEMENT

absolute motion
   go to line number: <line number>G
   go to last line: G (without a number)
   show
relative motion
   by character: arrow keys or hjkl (mneumonic: video game controls)
   by word: w forwards and b backwards
   by page: <ctrl>f forwards and <ctrl>b backwards
   beginning of line: 0
   end of line: $
search
   initial search: / forwards and ? backwards, followed by pattern
   find next: n forwards and N backwards
   find word under cursor: #
markers
   go to marked line: '<marker letter> (normal single quote)
   go to marked character: \`<marker letter> (back tick)

ACTIONS

insert new text
   insert before letter under cursor: i
   insert at beginning of line: I
   append after letter under cursor: a
   append after end of line: A
   open new line after current line: o
   open new line before current line: O
copy text (Yank)
   copy range: y (followed by motion to end of range)
   copy line: yy
delete text
   delete character: x (mneumonic: like typewriter strikeout)
   backspace character: X
   delete range: d (followed by motion to end of range)
   delete line: dd
paste text
   put after cursor: p
   put before cursor: P
replace text
   replace single character: r
   enter replace mode: R (text not typed over is unchanged)
change text (like delete followed by insert)
   change range: c (followed by motion to end of range)
   change to end of line: C
undo/redo
   undo last action: u
   undo all actions on current line: U
   redo last action: <ctrl>r
line formatting
   join lines: J
   reformat range to current textwidth: gq (followed by motion to end of range)
macros
   define macro: q<letter for macro>
   replay macro: @<letter for macro>

DEFINING RANGES

markers
   set marker: m<marker letter>
visual select (type again to cancel)
   visual select: v (followed by motion to end of range)
   line-mode visual select: V (followed by motion to end of range)
   column-mode visual select: <ctrl>v (followed by motion to end of range)

IMPORTANT COLON COMMANDS

help :help
write file :w (use :w! to force a write when you get a warning)
quit :q (use :q! to force a quit when you get a warning)
revert to saved :e!
edit another file :e <filename>
syntax highlighting :syntax on (or "off" to disable)
set options :set <option> (use ":set all" to see list of available options)
horizontally split window :sp
vertically split window :vsp
substitute :<range> s/<search pattern>/<replacement text>/

RANGE SYNTAX FOR COLON COMMANDS

'a,'z  = from marker a to marker z
.,$ = from current line to last line
1,. = from first line to last line
%  = whole file (short for "1,$")

USEFUL SET OPTIONS

enable paste mode (turn off default settings that make pasting act weird) :set paste
show command being constructed :set showcmd
show cursor line & column :set ruler
number lines :set number
ignore case on searches :set ignorecase (or "ic")
automatically break long lines :set textwidth=80 (or "tw")
convert from dos line-endings :set fileformat=unix (then save the file)