The Smell of Molten Projects in the Morning

Ed Nisley's Blog: Shop notes, electronics, firmware, machinery, 3D printing, laser cuttery, and curiosities. Contents: 100% human thinking, 0% AI slop.

Category: Software

General-purpose computers doing something specific

  • Arduino LiquidCrystal Library vs Old HD44780 LCD Controller

    I recently attached an ancient Optrex DM16117 LCD to an Arduino and discovered that the standard LiquidCrystal library routine wouldn’t initialize it properly. After turning on the power, the display would be blank. Hitting the Reset button did the trick, but that’s obviously not the right outcome.

    It turns out that initializing one of these widgets is trivially easy after you realize that the data sheet is required reading. If you do everything exactly right, then it works; get one step wrong, then the display might work most of the time, sorta-kinda, but most likely it won’t work, period.

    The catch is that there’s no such thing as a generic datasheet: what you must do depends on which version of the HD44780 controller lives on the specific LCD board in your hands and what oscillator frequency it’s using. The LiquidCrystal library seems to be written for a much newer and much faster version of the HD44780 than the one on my board, but, even so, the code may not be following all the rules.

    Optrex DMC16117 Initialization Sequence
    Optrex DMC16117 Initialization Sequence

    To begin…

    Fetch the Optrex DMC16117 datasheet, which includes the HD44780 timings for that family of LCD modules. There’s also a datasheet for just the Optrex LCD module itself, which isn’t quite what you want. You could get a bare Hitachi HD44780 datasheet, too, but it won’t have the timings you need.

    Pages 32 and 33 of the DMC16117 datasheet present the 8-bit and 4-bit initialization sequences. Given that no sane engineer uses the 8-bit interface, here’s the details of the 4-bit lashup.

    Two key points:

    • The first four transfers are not standard command sequences
    • The delays between transfers are not negotiable

    The starting assumption is that the LCD has not gone through the usual power-up initialization, perhaps because the supply voltage hasn’t risen at the proper rate. You could drive the LCD power directly from a microcontroller pin for a nice clean edge, but most designs really don’t have any pins to spare for that sort of nonsense: code is always cheaper than hardware (if you ignore non-recurring costs, that is, as many beancounters do).

    The Arduino LiquidCrystal library routine initialization sequence (in /opt/arduino/hardware/libraries/LiquidCrystal/LiquidCrystal.cpp) looks like this:

    command(0x28);  // function set: 4 bits, 1 line, 5x8 dots
command(0x0C);  // display control: turn display on, cursor off, no blinking
command(0x06);  // entry mode set: increment automatically, display shift, right shift
clear();

    The four-bit version of the command() function sends both nibbles of its parameter, high followed by low, which simply isn’t correct for the first few values the DMC16117 expects. Worse, the timing doesn’t follow the guidelines; there’s no delay at all between any of the outgoing values. Again, this is most likely due to the fact that LiquidCrystal was written for a newer version of the HD44780 chip.

    After a bit of fiddling around, I decided that the only solution was to create a new library routine based on LiquidCrystal with the proper delays and commands: LCD_Optrex. It might not work for newer LCDs, but at least it’ll play with what I have in my parts heap.

    Next, the gory details…

    Memo to Self: The protracted delay after the first Clear is absolutely vital!

  • Arduino IDE Race Condition

    Every now and again the Arduino IDE spits out an error message along the lines of “couldn’t determine program size” or simply fails to compile with no error message at all. The former is evidently harmless, but the latter can be truly annoying.

    The cause is described for Macs there as a race condition in the IDE on multi-core processors, with a patch that either partially fixes the problem or pushes it to a less-likely part of the code. That’s true on my system, as the error still occurs occasionally.

    How you apply it to Xubuntu 8.10: unzip the file to get Sizer.class, then copy that file to /usr/lib/jvm/java-6-sun-1.6.0.10/jre/lib/. That won’t be the right place for a different Xubuntu or different Java, so use locate rt.jar and plunk it into that directory.

    A less dramatic change seems to be setting build.verbose=true and upload.verbose=true in ~/.arduino/preferences.txt.

    In my case, both of those changes did bupkis.

    This is evidently an error of long standing, as it’s been discussed since about Arduino 11. I’m currently at 15 and it seems that patch will be in the next version of the IDE.

  • Arduino Push-Pull PWM

    Push-Pull Drive: OC1A top, OC1B bottom
    Push-Pull PWM Drive: OC1A top, OC1B bottom

    Most of the time you need just a single PWM output, but when you’re driving a transformer (or some such) and need twice the primary voltage, you can use a pair of PWM outputs in push-pull mode to get twice the output voltage.

    A single PWM output varies between 0 and +5 V (well, Vcc: adjust as needed), so the peak-to-peak value is 5 V. Drive a transformer from two out-of-phase PWM outputs, such that one is high while the other is low, and the transformer sees a voltage of 5 V one way and 5 V the other, for a net 10 V peak-to-peak excursion.

    A 50% duty cycle will keep DC out of the primary winding, but a blocking capacitor is always a good idea with software-controlled hardware. A primary winding with one PWM output stuck high and the other stuck low is a short circuit that won’t do your output drivers or power supply any good at all.

    An Arduino (ATMega168 and relatives) can do this without any additional circuitry if you meddle with the default PWM firmware setup. You must use a related pair of PWM outputs that share an internal timer; I used PWM 9 and 10.

    #define PIN_PRI_A   9    // OCR1A - high-active primary drive
#define PIN_PRI_B   10   // OCR1B - low-active primary drive

    I set push-pull mode as a compile-time option, but you can change it on the fly.

    #define PUSH_PULL   true // false = OCR1A only, true = OCR1A + OCR1B

    The timer tick rate depends on what you’re trying accomplish. I needed something between 10 & 50 kHz, so I set the prescaler to 1 to get decent resolution: 62.5 ns.

    // Times in microseconds, converted to timer ticks
//  ticks depend on 16 MHz clock, so ticks = 62.5 ns
//  prescaler can divide that, but we're running fast enough to not need it

#define TIMER1_PRESCALE   1     // clock prescaler value
#define TCCR1B_CS20       0x01  // CS2:0 bits = prescaler selection

    Phase-Frequency Correct mode (WGM1 = 8) runs at half-speed (it counts both up and down in each cycle), so the number of ticks is half what you’d expect. You could use Fast PWM mode or anything else, as long as you get the counts right; see page 133 of the Fine Manual.

    // Phase-freq Correct timer mode -> runs at half the usual rate

#define PERIOD_US   60
#define PERIOD_TICK (microsecondsToClockCycles(PERIOD_US / 2) / TIMER1_PRESCALE)

    The phase of the PWM outputs comes from the Compare Output Mode register settings. Normally the output pin goes high when the PWM count resets to zero and goes low when it passes the duty cycle setting, but you can flip that around. The key bits are COM1A and COM1B in TCCR1A, as documented on page 131.

    As always, it’s easiest to let the Arduino firmware do its usual setup, then mercilessly bash the timer configuration registers…

    // Configure Timer 1 for Freq-Phase Correct PWM
//   Timer 1 + output on OC1A, chip pin 15, Arduino PWM9
//   Timer 1 - output on OC1B, chip pin 16, Arduino PWM10

 analogWrite(PIN_PRI_A,128);    // let Arduino setup do its thing
 analogWrite(PIN_PRI_B,128);

 TCCR1B = 0x00;                 // stop Timer1 clock for register updates

// Clear OC1A on match, P-F Corr PWM Mode: lower WGM1x = 00
 TCCR1A = 0x80 | 0x00;

// If push-pull drive, set OC1B on match
#if PUSH_PULL
 TCCR1A |= 0x30;
#endif

 ICR1 = PERIOD_TICKS;           // PWM period
 OCR1A = PERIOD_TICKS / 2;      // ON duration = drive pulse width = 50% duty cycle
 OCR1B = PERIOD_TICKS / 2;      //  ditto - use separate load due to temp buffer reg
 TCNT1 = OCR1A - 1;             // force immediate OCR1x compare on next tick

// upper WGM1x = 10, Clock Sel = prescaler, start Timer 1 running
 TCCR1B = 0x10 | TCCR1B_CS20;

    And now you’ll see PWM9 and PWM10 running in opposition!

    Memo to Self: remember that “true” does not equal “TRUE” in the Arduino realm.

  • Recovering JPG Images From Damaged Flash Memory

    My DSC-F717 just crashed as I took a picture in the Basement Laboratory; it stalled while writing a picture to the memory stick. This camera is known to have problems with its ribbon cables and I’ve fixed it a few times before, but right now I have other things to do. Who knows? Maybe it’s a different problem altogether.

    Thus, the pertinent question is how to grab the image files from the Memory Stick, even with a damaged filesystem. This trick will work with any memory device, so it’s not just a Memory Stick thing.

    The camera crashed with the Memory Stick activity LED stuck on. Turning the camera off didn’t work: it jammed in the power-down sequence. So I poked the Reset button, which snapped the camera back to 1 January 2002, just after midnight. Alas, it now couldn’t read the Memory Stick, which meant either the filesystem is pooched or the camera’s card socket has gone bad again.

    Mmm, do you know where your camera’s Reset button is? It might not even have one, in which case you just yank the battery. If you can yank the battery, that is.

    Anyhow…

    With the camera turned off: extract the memory card, poke it into a suitable USB card reader, and poke that into your PC (running Linux, of course).

    If the filesystem isn’t too badly damaged, you’ll probably get a popup asking what to do with the new drive (the memory card). Dismiss that, as you don’t want anything writing to the memory without your explicit permission, which you won’t give.

    Figure out which device corresponds to the card and which partition to use:

    dmesg | tail
[29846.600524] sd 5:0:0:2: [sdd] 1947648 512-byte hardware sectors (997 MB)
[29846.606022] sd 5:0:0:2: [sdd] Write Protect is off
[29846.606030] sd 5:0:0:2: [sdd] Mode Sense: 23 00 00 00
[29846.606033] sd 5:0:0:2: [sdd] Assuming drive cache: write through
[29846.608748]  sdd: sdd1

    In this case, the Memory Stick was intact enough to have a valid partition table, so it could be successfully mounted. However, you don’t want any attempt to write the data, just to keep a bad situation from getting worse. You do that by mounting the device read-only:

    sudo mount -o ro /dev/sdd1 /mnt/part

    You’ll need a suitable mount point; I created /mnt/part early on to hold manually mounted disk partitions.

    The FAT filesystem seemed to point to valid files, but attempting to display them didn’t work. So unmount the device before proceding:

    sudo umount /dev/sdd1

    Switch to /tmp or anywhere you have enough elbow room for a few files the size of the memory card.

    Run dd to make a bit-for-bit copy of the entire drive:

    sudo dd if=/dev/sdd of=memstick.bin bs=1M

    The bs=1M should dramatically speed up the transfer; the default is, IIRC, 512 bytes.

    After this, everything you do uses memstick.bin, not the memory card itself. If you’re paranoid like me, you’ll make a copy of the file before you do anything hazardous. You could even make two copies directly from the memory card and compare them, which might be instructive.

    To find your precious JPG images among the rubble, look for their magic Exif headers:

    strings -t x memstick.bin | grep Exif
  68006 Exif
 258006 Exif
 680006 Exif
 848006 Exif
 a18006 Exif
 bf0006 Exif
 e00006 Exif
 fe8006 Exif
11e8006 Exif
1420006 Exif
... snippage ...

    Because the strings search ignores the (presumably broken) FAT directory structure, you’ll find all the image files on the drive, whether or not they’ve been deleted, partially overwritten, or whatever. This is great for forensics and terrible if you’ve got something to hide. You have been warned.

    The -t x parameter returns the string’s starting offset in hex, which you need to find the actual starting offset of the file: it’s 6 bytes lower! Your mileage may vary, so be prepared to fiddle around a bit.

    You could write a bunch of code to parse the JPG header and extract exactly the number of bytes required, but this is no time for subtle gestures. I just yank out whatever the largest possible image file could be, because image-processing programs only process the valid stuff anyway. So, knowing that this camera produces images in the 2.4 MB range, this extracts the first image:

    dd if=memstick.bin of=image.jpg bs=$((16#1000)) count=1K skip=$((16#68))

    The $(( … )) notation evaluates the numeric expression within and the 16#… notation expresses a hexadecimal value.

    Soooo, bs=$((16#1000)) says that the blocksize is 4096 bytes, which you can deduce from the fact that all the Exif headers start 6 bytes from a multiple of 0x1000. Again, your camera may do things differently, but the general notion should get you started.

    If you’re fussy, you’ll note that the headers are actually on multiples of 0x8000 bytes, but using 0x1000 means you can read the high-order digits right off the strings dump. Why make things more complicated than necessary?

    Given a 4K blocksize, count=1K extracts a 4MB chunk: 1024 blocks of 4096 bytes each. That’s larger than the largest possible image file from this camera; adjust it to suit whatever you expect to find. Don’t be stingy, OK?

    The skip=$((16#68)) says to begin extracting data 0x68 blocks into memstick.bin. You read that value directly from the strings output, which is easy enough.

    You could write a tidy Bash script to eat the strings values and spit out the corresponding file chunks. I had few enough images this time to just do it manually, which beats having to debug some code…

    Good luck!

  • Terabyte Backup for the Backup

    Now that terabyte drives sell for under 100 bucks, there’s no longer any reason to worry about backup space. Just do it, OK?

    My file server runs a daily backup (using rsnapshot, about which more later) just around midnight, copying all the changed files to an external 500 GB USB drive. On the first of each month, it sets aside the current daily backup as a monthly set, so I have a month of days and a year of months.

    Roughly once a quarter, I copy the contents of that drive to another drive, empty the file system, and start over again.

    Right now there’s about 380 GB of files on the server, but rsnapshot maintains only one copy of each changed file on the backup drive and we typically change only a few tens of megabytes each day, sooo a 500 GB drive doesn’t fill up nearly as fast as you might think.

    Our daughter’s doing a science fair project involving ballistics and video recording, so she recently accumulated 36 GB of video files in short order… and re-captured the entire set several times. Of course the external drive filled up, so it’s time for the swap.

    Recently I picked up a 1 TB SATA drive and it’s also time to document that process.

    You will, of course, have already set up SSH and added your public key to that box, so you can do this from your Comfy Chair rather than huddling in the basement. You’ll also use screen so you can disconnect from the box while letting the session run overnight.

    Plug the drive into a SATA-to-USB converter, which will most likely pop up a cheerful dialog asking what you want to do with the FAT/NTFS formatted empty drive. Dismiss that; you’re going to blow it away and use ext2.

    Find out which drive it is

    dmesg | tail
    [25041.488000] sd 8:0:0:0: [sdc] 1953525168 512-byte hardware sectors (1000205 MB)
[25041.488000] sd 8:0:0:0: [sdc] Write Protect is off
[25041.488000] sd 8:0:0:0: [sdc] Mode Sense: 00 38 00 00
[25041.488000] sd 8:0:0:0: [sdc] Assuming drive cache: write through
[25041.488000]  sdc: sdc1

    Unmount the drive

    sudo umount /dev/sdc1

    Create an empty ext2 filesystem

    sudo mke2fs -v -m 0 -L Backup1T /dev/sdc1

    The -v lets you watch the lengthy proceedings. The -m 0 eliminates the normal 5% of the capacity reserved for root; you won’t be running this as a normal system drive and won’t need emergency capacity for logs and suchlike. The -L gives it a useful name.

    Create a mount point and mount the new filesystem

    sudo mkdir /mnt/part
sudo mount /dev/sdc1 /mnt/part

    You may have to fight off another automounter intervention in there somewhere.

    The existing backup drive is in /etc/fstab for easy mounting by the the rsync script. Because it’s a USB drive, I used its UUID name rather than a /dev name that depends on what else might be plugged in at the time. To find that out

    ll /dev/disk/by-uuid/
 ... snippage ...
lrwxrwxrwx 1 root root 10 2009-03-17 09:54 fedcdb1c-ec6e-4edc-be35-22915c82e46a -> ../../sdd1

    So then this gibberish belongs in /etc/fstab

    UUID=fedcdb1c-ec6e-4edc-be35-22915c82e46a /mnt/backup ext3 defaults,noatime,noauto,rw,nodev,noexec,nosuid 0 0

    Mount the existing backup drive and dump its contents to the new one:

    sudo mount /mnt/backup
sudo rsync -aHu --progress --bwlimit=10000 --exclude=".Trash-1**" /mnt/backup/snapshots /mnt/part

    You need sudo to get access to files from other users.

    Rsync is the right hammer for this job; don’t use cp.

    The -a preserves all the timestamps & attributes & owners, which is obviously a Good Thing.

    The -H preserves hard links, which is what rsnapshot uses to maintain one physical copy in multiple snapshot directories; if you forget this, you’ll get one physical copy for every snapshot directory and run out of space on that 1 TB drive almost immediately.

    The -u does an update, which is really helpful if you must interrupt the process in midstream. Trust me on this, you will interrupt it from time to time.

    You will also want to watch the proceedings, which justifies –progress. There’s a torrent of screen output, but it’s mostly just comfort noise.

    The key parameter is –bwlimit=10000, which throttles the transfer down to a reasonable level and leaves some CPU available for normal use. Unthrottled USB-to-USB transfer ticks along at about 15 MB/s on my system, which tends to make normal file serving and printing rather sluggish. Your mileage will vary; I use –bwlimit=5000 during the day.

    You probably want to exclude the desktop trash files, which is what the –exclude=”.Trash-1**” accomplishes. If your user IDs aren’t in the 1000 range, adjust that accordingly.

    How long will it take? Figure 500 GB / 10 MB/s = 50 k seconds = 14 hours.

    That’s why you use screen: so you can shut down your Comfy Chair system overnight and get some shut-eye while rsync continues to tick along.

    When that’s all done, compare the results to see how many errors happen while shuffling the data around:

    sudo diff -q -r --speed-large-files /mnt/backup/snapshots/ /mnt/part/snapshots/ | tee > diff.log

    The sudo gives diff access to all those files, but the tee just records what it gets into a file owned by me.

    You’ll want to do that overnight, as it really hammers the server for CPU and I/O bandwidth. Batch is back!

    That was the theory, anyway, but it turned out the new drive consistently disconnected during the diff and then emerged with no filesystem. A definite disappointment after a half-day copy operation and something of a surprise given that diff is a read-only operation.

    A considerable amount of fiddling showed that running USB-to-USB copies simply didn’t work, even if the drives were on different inside-the-PC USB controllers, with failures occurring around a third of a terabyte. So, rather than debugging that mess, I wound up making copies directly from the file server’s internal drives, which ran perfectly but also ignored the deep history on the backup drive.

    But, eh, I’ve been stashing a drive in the safe deposit box for the last few years, so there should be enough history to go around…

  • Vital Browser Addition: Readability

    Short version: Go there, read about Readability, set it up, and browse happily ever after.

    Longer version: Readability chops away all the overdesigned Webbish crap around the text you want to read, reformats it in a single block, and lets you read without distractions.

    My configuration:

    • Style: Novel
    • Size: Large
    • Margin: Narrow

    I put it as the top bookmark in the sidebar, where I hit it rather often.

    It also works well for printing: nytimes.com articles now print quite legibly in four-up mode, which saves a ton o’ paper.

    Bonus: you can even print those pesky blogs that don’t print in Firefox.

    Minor disadvantage: if you want to print a whole article, you must get all the text on a single page. Some blogs / news outlets don’t let you do that, for reasons that should be obvious when you consider how nice Readability is.

    Just do it.

    You should, of course, be using the Firefox Adblock Plus Add-On to quiet your browsing even more. There used to be an ethical question about using ad-supported sites while running ad-blocking software, but that seems to have died out with the advent of pop-up/pop-under ads, animated GIFs, and relentless Flash junk.

  • Linux Install Tweaks: Firefox

    I wish there was a way to bulk-install a list of add-ons, rather than futzing around installing them one-by-one. On the other hand, this way you’re sure to get the latest-and-greatest, while not attempting to install anything that’s obsolete.

    Copy passwords & suchlike from your previous installation into ~/.mozilla/firefox/whatever

    • key3.db
    • signon3.txt

    Install add-ons:

    • Adblock Plus
    • GreaseMonkey — then get GoogleMonkeyR for two-up search results
    • Google Gears — for wordpress blogging
    • Image Zoom
    • Tab Focus — 250 ms delay
    • Print Context Menu
    • FireGestures
    • Linkification
    • PDF Download
    • Brief

    Install themes

    • Microfox
    • Littlefox
    • Classic Compact

    I like Microfox: it doesn’t waste vertical space on foo-foos.

    Kill off Flash cookies and disable all the sharing and enabling and local storage: does anyone really want Flash apps to use the microphone & camera by default? This technique is not obvious and you must do it for every user and every browser on each system.

    The Adobe Knowledgebase article is here:

    http://kb.adobe.com/selfservice/viewContent.do?externalId=52697ee8

    The actual Flash control panel is here:

    http://www.macromedia.com/support/documentation/en/flashplayer/help/settings_manager02.html

    You should also search on the obvious keywords for more info.

    Exit Firefox, fire it up again, do the obvious tweakage, and you’re set!