The Smell of Molten Projects in the Morning

Ed Nisley's Blog: Shop notes, electronics, firmware, machinery, 3D printing, laser cuttery, and curiosities. Contents: 100% human thinking, 0% AI slop.

Category: Software

General-purpose computers doing something specific

  • Displaying Variables in Gnuplot

    Sample plot with regression line & variables
    Sample plot with curve-fit line & variables

    Gnuplot can do curve fitting (of all kinds) and parks the coefficients in variables. In general, you’d like to display those values on the final plot for later reference…

    The trick is using the sprintf() function, which behaves largely like the C version, to insert the variable into a formatted string for use in the label command.

    I drive Gnuplot with shell scripts, which simplifies introducing parameters & suchlike. That’s conspicuous by its absence here, but when you need it, you need it bad.

    The script to generate that plot looks like this, with some key points in the highlighted lines:

    #!/bin/sh
    export GDFONTPATH="/usr/share/fonts/truetype/msttcorefonts/"
    gnuplot << EOF
    set term png font "arialbd.ttf" 18 size 950,600
    set output "Calibration Curve - Full.png"
    set title "Calibration Curve - Full"
    set key noautotitles
    unset mouse
    set bmargin 4
    set grid xtics ytics
    set xlabel "10^5/ADC"
    set format x "%3.0f"
    set ylabel "Resistance - Ohm"
    set format y "%3.0f"
    set yrange [0:100]
    set datafile separator "\t"
    f(x) = m*x + c
    fit f(x) "Measurements/Calibration.csv" using 3:1 via m,c
    set label 1 sprintf("m = %3.4f",m) at 510,75 font "arialbd,18"
    set label 2 sprintf("c = %3.4f",c) at 510,70 font "arialbd,18"
    plot    \
     "Measurements/Calibration.csv" \
     using 3:1 with linespoints lt 3 lw 3 pt 3 , \
    f(x) lt 4 lw 2 
    EOF
    

    The dataset for that plot is tucked into the obvious file and looks like this, with tabs between the columns:

    # ESR Calibration Curve
    # Resistance    ADC Decimal    Reciprocal
    0.0    492    203
    0.1    489    204
    1.0    461    217
    1.2    456    219
    1.5    447    224
    1.8    440    227
    2.0    432    231
    2.3    428    234
    2.4    423    236
    2.7    414    242
    3.3    400    250
    3.8    387    258
    4.3    378    265
    4.7    367    272
    5.0    360    278
    5.5    350    286
    6.3    332    301
    6.7    329    304
    8.1    306    327
    9.1    293    341
    9.9    284    352
    21.0    182    549
    33.0    126    794
    47.0    90    1111
    67.0    60    1667
    73.0    54    1852
    83.0    47    2128
    92.0    41    2439

    There is no denying that a straight line is not the best fit to that dataset, but that’s not the point.

    Memo to Self: the set label commands go between the fit and the plot. Remember to add the f(x) to the plot function…

  • Why Friends Don’t Let Friends Use Windows: Torpig

    For those of you still using Windows, here’s a sobering look at why you shouldn’t: an analysis of the Torpig botnet by an academic group that managed to take over its command & control structure for a few days.

    The report is tech-heavy, but well worth the effort to plow through.

    Here are some of the high points…

    Why do the bad guys do this? It’s all about the money, honey:

    In ten days, Torpig obtained the credentials of 8,310 accounts at 410 different institutions.

    … we extracted 1,660 unique credit and debit card numbers from our
    collected data.

    Does an antivirus program help?

    Torpig has been distributed to its victims as part of Mebroot. Mebroot is a rootkit that takes control of a machine by replacing the system’s Master Boot Record (MBR). This allows Mebroot to be executed at boot time, before the operating system is loaded, and to remain undetected by most anti-virus tools

    In these attacks, web pages on legitimate but vulnerable web sites are modified with the inclusion of HTML tags that cause the victim’s browser to request JavaScript code from a[nother] web site under control of the attackers. This JavaScript code launches a number of exploits against the browser or some of its components, such as ActiveX controls and plugins. If any exploit is successful, an executable is downloaded from the drive-by-download server to the victim machine, and it is executed.

    What happens next?

    Mebroot injects these modules […] into a number of applications. These applications include the Service Control Manager (services.exe), the file manager, and 29 other popular applications, such as web browsers (e.g., Internet Explorer, Firefox, Opera), FTP clients (Leech-FTP, CuteFTP), email clients (e.g., Thunderbird, Outlook, Eudora), instant messengers (e.g., Skype, ICQ), and system programs (e.g., the command line interpreter cmd.exe). After the injection, Torpig can inspect all the data handled by these programs and identify and store interesting pieces of information, such as credentials for online accounts and stored passwords.

    If you think hiding behind a firewall router will save you, you’re wrong:

    By looking at the IP addresses in the Torpig headers we are able to determine that 144,236 (78.9%) of the infected machines were behind a NAT, VPN, proxy, or firewall.

    If you think you’ve got a secure password, you’re wrong:

    Torpig bots stole 297,962 unique credentials (i.e., username and password pairs), sent by 52,540 different Torpig-infected machines over the ten days we controlled the botnet

    If you think a separate password manager will save you, you’re wrong.

    It is also interesting to observe that 38% of the credentials stolen by Torpig were obtained from the password manager of browsers, rather than by intercepting an actual login session.

    Somewhat more info on Mebroot from F-Secure.

    Remember, the virus / worm / Trojan / botnet attacks you read about all the time only affect Windows machines. Linux isn’t invulnerable, but it’s certainly safer right now. If you’re running Windows, it’s only a matter of time until your PC is not your own, no matter how smart you think you are.

    If you have one or two must-gotta-use Windows programs, set up a dedicated Token Windows Box and use it only for those programs. Network it (behind a firewall) if you like, but don’t do any email / Web browsing / messaging / VOIP on it. Just Say No!

    For everything else, run some version of Linux. It’ll do what you need to get done with less hassle and far less risk. It’s free for the download, free for the installation, and includes all the functions you’re used to paying money for. Just Do It!

    If you think using Linux is too much of a hassle, imagine what putting your finances back together will be like. Remember, the bad guys will steal everything you’ve ever put on your PC, destroy your identity, and never get caught.

    Now you know… why are you still stalling?

  • Arduino Library for (Old?) Optrex DMC-family LCDs

    Having determined that the existing Arduino LiquidCrystal library routine wouldn’t work for the Optrex DMC-16117 character LCD in my parts heap, I decided to modify it to meet the data and timing requirements mentioned in the datasheet. This is sufficiently slow and old that it should work for contemporary displays built around the Hitachi HD44780 and its ilk, but I certainly haven’t tested it!

    The straight dope on building an Arduino library from scratch is there, but there’s no need to work from First Principles here.

    Start by copying the library files (this FLOSS stuff is wonderful that way), renaming them, and changing all the LiquidCrystal strings to LCD_Optrex:

    cd /opt/arduino/hardware/libraries/
    cp -a LiquidCrystal LCD_Optrex
    cd LCD_Optrex
    rm LiquidCrystal.o
    rename 's/LiquidCrystal/LCD_Optrex/' LiquidCrystal.*
    for f in LCD* k*; do sed -i 's/LiquidCrystal/LCD_Optrex/g' $f; done
    cd examples
    for f in * ; do sed -i 's/LiquidCrystal/LCD_Optrex/g' $f/$f.pde
    cd -
    

    You could do that by hand with an editor if you prefer.

    Depending on how you’ve installed the Arduino files, you may need a sudo to make that work. Better, perhaps, to tweak the permissions for (at least) the LCD_Optrex directory & files therein to grant yourself write access.

    I created a sendraw4() function to send a single 4-bit nibble during the startup sequence, so add that to the private section of LCD_Optrex.h:

    private:
    void send(uint8_t, uint8_t);
    void sendraw4(uint8_t);
    

    The new code is in LCD_Optrex is shamelessly adapted from the existing send() function, minus the mode selection and 8-bit stuff:

    void LCD_Optrex::sendraw4(uint8_t value) {
      digitalWrite(_rs_pin, LOW);
      digitalWrite(_rw_pin, LOW);
    
      for (int i = 0; i < 4; i++) {
        digitalWrite(_data_pins[i], (value >> (i + 4)) & 0x01);
      }
    
      digitalWrite(_enable_pin, HIGH);
      digitalWrite(_enable_pin, LOW);
    }
    

    If I were doing this from scratch, I’d use d7 through d4 rather than d3 through d0 to match the datasheet, but that’s a stylin’ thing.

    Replace the existing LCD panel setup code with an exact mapping of the datasheet’s procedure. For the 4-bit setup, it goes a little something like this:

    delayMicroseconds(16000);       // mandatory delay for Vcc stabilization
    sendraw4(0x30);                 // set 8-bit mode (yes, it's true!)
    delayMicroseconds(5000);        // mandatory delay
    sendraw4(0x30);
    delayMicroseconds(200);
    sendraw4(0x30);
    delayMicroseconds(40);          // command delay
    sendraw4(0x20);                 // finally set 4-bit mode
    delayMicroseconds(40);          // command delay
    
    command(0x28);            // 4-bit, 2-line, 5x7 char set
    command(0x08);            // display off
    command(0x01);            // clear display
    delayMicroseconds(16000);
    command(0x06);            // increment, no shift
    command(0x0c);            // display on, no cursor, no blink
    

    It seems you cannot use the delay() function in the constructor, as interrupts and suchlike aren’t active. The delayMicroseconds() function disables & enables interrupts; I don’t know if that is a Bad Thing or not.

    The 8-bit initialization code, which I haven’t tested, doesn’t need the sendraw4() function, but does need the same alterations. Apart from enabling 4-bit mode, of course.

    Various commands have different timing requirements, as shown on page 39 of the DMC16117 datasheet. Add a delayMicroseconds(16000); to the clear() and home() functions, then add delayMicroseconds(40); to the send() function, like this:

    void LCD_Optrex::clear()
    {
      command(0x01);  // clear display, set cursor position to zero
      delayMicroseconds(16000);
    }
    
    Optrex DMC16117 Instruction Timing
    Optrex DMC16117 Instruction Timing

    With all that in place, fire up the Arduino IDE and compile one of the example programs. That will build the LCD_Optrex.o file, too. If you have such a display, either wire it up as indicated or change the example code to match your connections.

    What should happen is that the LCD should initialize correctly under all conditions… how’s that for anticlimactic?

    Here’s an OpenOffice document with LCD_Optrex.h, LCD_Optrex.cpp, and examples.txt all in one lump: LCD_Optrex Library Files.odt. Save each section as a separate flat-ASCII text file with the appropriate name in the right spot and you’re in business. I’d present a ZIP file, but WordPress isn’t up to that.

    Memo to Self: A function to write a 16-character string to the stupid 16-character DMC16117 which has a single row that’s addressed as two 8-character lines would be nice. That requires keeping track of the current cursor position, which could be tricky. Maybe I should just scrap those suckers out?

  • Arduino LiquidCrystal Library vs Old HD44780 LCD Controller

    I recently attached an ancient Optrex DM16117 LCD to an Arduino and discovered that the standard LiquidCrystal library routine wouldn’t initialize it properly. After turning on the power, the display would be blank. Hitting the Reset button did the trick, but that’s obviously not the right outcome.

    It turns out that initializing one of these widgets is trivially easy after you realize that the data sheet is required reading. If you do everything exactly right, then it works; get one step wrong, then the display might work most of the time, sorta-kinda, but most likely it won’t work, period.

    The catch is that there’s no such thing as a generic datasheet: what you must do depends on which version of the HD44780 controller lives on the specific LCD board in your hands and what oscillator frequency it’s using. The LiquidCrystal library seems to be written for a much newer and much faster version of the HD44780 than the one on my board, but, even so, the code may not be following all the rules.

    Optrex DMC16117 Initialization Sequence
    Optrex DMC16117 Initialization Sequence

    To begin…

    Fetch the Optrex DMC16117 datasheet, which includes the HD44780 timings for that family of LCD modules. There’s also a datasheet for just the Optrex LCD module itself, which isn’t quite what you want. You could get a bare Hitachi HD44780 datasheet, too, but it won’t have the timings you need.

    Pages 32 and 33 of the DMC16117 datasheet present the 8-bit and 4-bit initialization sequences. Given that no sane engineer uses the 8-bit interface, here’s the details of the 4-bit lashup.

    Two key points:

    • The first four transfers are not standard command sequences
    • The delays between transfers are not negotiable

    The starting assumption is that the LCD has not gone through the usual power-up initialization, perhaps because the supply voltage hasn’t risen at the proper rate. You could drive the LCD power directly from a microcontroller pin for a nice clean edge, but most designs really don’t have any pins to spare for that sort of nonsense: code is always cheaper than hardware (if you ignore non-recurring costs, that is, as many beancounters do).

    The Arduino LiquidCrystal library routine initialization sequence (in /opt/arduino/hardware/libraries/LiquidCrystal/LiquidCrystal.cpp) looks like this:

    command(0x28);  // function set: 4 bits, 1 line, 5x8 dots
    command(0x0C);  // display control: turn display on, cursor off, no blinking
    command(0x06);  // entry mode set: increment automatically, display shift, right shift
    clear();
    

    The four-bit version of the command() function sends both nibbles of its parameter, high followed by low, which simply isn’t correct for the first few values the DMC16117 expects. Worse, the timing doesn’t follow the guidelines; there’s no delay at all between any of the outgoing values. Again, this is most likely due to the fact that LiquidCrystal was written for a newer version of the HD44780 chip.

    After a bit of fiddling around, I decided that the only solution was to create a new library routine based on LiquidCrystal with the proper delays and commands: LCD_Optrex. It might not work for newer LCDs, but at least it’ll play with what I have in my parts heap.

    Next, the gory details…

    Memo to Self: The protracted delay after the first Clear is absolutely vital!

  • Arduino IDE Race Condition

    Every now and again the Arduino IDE spits out an error message along the lines of “couldn’t determine program size” or simply fails to compile with no error message at all. The former is evidently harmless, but the latter can be truly annoying.

    The cause is described for Macs there as a race condition in the IDE on multi-core processors, with a patch that either partially fixes the problem or pushes it to a less-likely part of the code. That’s true on my system, as the error still occurs occasionally.

    How you apply it to Xubuntu 8.10: unzip the file to get Sizer.class, then copy that file to /usr/lib/jvm/java-6-sun-1.6.0.10/jre/lib/. That won’t be the right place for a different Xubuntu or different Java, so use locate rt.jar and plunk it into that directory.

    A less dramatic change seems to be setting build.verbose=true and upload.verbose=true in ~/.arduino/preferences.txt.

    In my case, both of those changes did bupkis.

    This is evidently an error of long standing, as it’s been discussed since about Arduino 11. I’m currently at 15 and it seems that patch will be in the next version of the IDE.

  • Arduino Push-Pull PWM

    Push-Pull Drive: OC1A top, OC1B bottom
    Push-Pull PWM Drive: OC1A top, OC1B bottom

    Most of the time you need just a single PWM output, but when you’re driving a transformer (or some such) and need twice the primary voltage, you can use a pair of PWM outputs in push-pull mode to get twice the output voltage.

    A single PWM output varies between 0 and +5 V (well, Vcc: adjust as needed), so the peak-to-peak value is 5 V. Drive a transformer from two out-of-phase PWM outputs, such that one is high while the other is low, and the transformer sees a voltage of 5 V one way and 5 V the other, for a net 10 V peak-to-peak excursion.

    A 50% duty cycle will keep DC out of the primary winding, but a blocking capacitor is always a good idea with software-controlled hardware. A primary winding with one PWM output stuck high and the other stuck low is a short circuit that won’t do your output drivers or power supply any good at all.

    An Arduino (ATMega168 and relatives) can do this without any additional circuitry if you meddle with the default PWM firmware setup. You must use a related pair of PWM outputs that share an internal timer; I used PWM 9 and 10.

    #define PIN_PRI_A   9    // OCR1A - high-active primary drive
    #define PIN_PRI_B   10   // OCR1B - low-active primary drive
    

    I set push-pull mode as a compile-time option, but you can change it on the fly.

    #define PUSH_PULL   true // false = OCR1A only, true = OCR1A + OCR1B
    

    The timer tick rate depends on what you’re trying accomplish. I needed something between 10 & 50 kHz, so I set the prescaler to 1 to get decent resolution: 62.5 ns.

    // Times in microseconds, converted to timer ticks
    //  ticks depend on 16 MHz clock, so ticks = 62.5 ns
    //  prescaler can divide that, but we're running fast enough to not need it
    
    #define TIMER1_PRESCALE   1     // clock prescaler value
    #define TCCR1B_CS20       0x01  // CS2:0 bits = prescaler selection
    

    Phase-Frequency Correct mode (WGM1 = 8) runs at half-speed (it counts both up and down in each cycle), so the number of ticks is half what you’d expect. You could use Fast PWM mode or anything else, as long as you get the counts right; see page 133 of the Fine Manual.

    // Phase-freq Correct timer mode -> runs at half the usual rate
    
    #define PERIOD_US   60
    #define PERIOD_TICK (microsecondsToClockCycles(PERIOD_US / 2) / TIMER1_PRESCALE)
    

    The phase of the PWM outputs comes from the Compare Output Mode register settings. Normally the output pin goes high when the PWM count resets to zero and goes low when it passes the duty cycle setting, but you can flip that around. The key bits are COM1A and COM1B in TCCR1A, as documented on page 131.

    As always, it’s easiest to let the Arduino firmware do its usual setup, then mercilessly bash the timer configuration registers…

    // Configure Timer 1 for Freq-Phase Correct PWM
    //   Timer 1 + output on OC1A, chip pin 15, Arduino PWM9
    //   Timer 1 - output on OC1B, chip pin 16, Arduino PWM10
    
     analogWrite(PIN_PRI_A,128);    // let Arduino setup do its thing
     analogWrite(PIN_PRI_B,128);
    
     TCCR1B = 0x00;                 // stop Timer1 clock for register updates
    
    // Clear OC1A on match, P-F Corr PWM Mode: lower WGM1x = 00
     TCCR1A = 0x80 | 0x00;
    
    // If push-pull drive, set OC1B on match
    #if PUSH_PULL
     TCCR1A |= 0x30;
    #endif
    
     ICR1 = PERIOD_TICKS;           // PWM period
     OCR1A = PERIOD_TICKS / 2;      // ON duration = drive pulse width = 50% duty cycle
     OCR1B = PERIOD_TICKS / 2;      //  ditto - use separate load due to temp buffer reg
     TCNT1 = OCR1A - 1;             // force immediate OCR1x compare on next tick
    
    // upper WGM1x = 10, Clock Sel = prescaler, start Timer 1 running
     TCCR1B = 0x10 | TCCR1B_CS20;
    

    And now you’ll see PWM9 and PWM10 running in opposition!

    Memo to Self: remember that “true” does not equal “TRUE” in the Arduino realm.

  • Recovering JPG Images From Damaged Flash Memory

    My DSC-F717 just crashed as I took a picture in the Basement Laboratory; it stalled while writing a picture to the memory stick. This camera is known to have problems with its ribbon cables and I’ve fixed it a few times before, but right now I have other things to do. Who knows? Maybe it’s a different problem altogether.

    Thus, the pertinent question is how to grab the image files from the Memory Stick, even with a damaged filesystem. This trick will work with any memory device, so it’s not just a Memory Stick thing.

    The camera crashed with the Memory Stick activity LED stuck on. Turning the camera off didn’t work: it jammed in the power-down sequence. So I poked the Reset button, which snapped the camera back to 1 January 2002, just after midnight. Alas, it now couldn’t read the Memory Stick, which meant either the filesystem is pooched or the camera’s card socket has gone bad again.

    Mmm, do you know where your camera’s Reset button is? It might not even have one, in which case you just yank the battery. If you can yank the battery, that is.

    Anyhow…

    With the camera turned off: extract the memory card, poke it into a suitable USB card reader, and poke that into your PC (running Linux, of course).

    If the filesystem isn’t too badly damaged, you’ll probably get a popup asking what to do with the new drive (the memory card). Dismiss that, as you don’t want anything writing to the memory without your explicit permission, which you won’t give.

    Figure out which device corresponds to the card and which partition to use:

    dmesg | tail
    [29846.600524] sd 5:0:0:2: [sdd] 1947648 512-byte hardware sectors (997 MB)
    [29846.606022] sd 5:0:0:2: [sdd] Write Protect is off
    [29846.606030] sd 5:0:0:2: [sdd] Mode Sense: 23 00 00 00
    [29846.606033] sd 5:0:0:2: [sdd] Assuming drive cache: write through
    [29846.608748]  sdd: sdd1
    

    In this case, the Memory Stick was intact enough to have a valid partition table, so it could be successfully mounted. However, you don’t want any attempt to write the data, just to keep a bad situation from getting worse. You do that by mounting the device read-only:

    sudo mount -o ro /dev/sdd1 /mnt/part

    You’ll need a suitable mount point; I created /mnt/part early on to hold manually mounted disk partitions.

    The FAT filesystem seemed to point to valid files, but attempting to display them didn’t work. So unmount the device before proceding:

    sudo umount /dev/sdd1

    Switch to /tmp or anywhere you have enough elbow room for a few files the size of the memory card.

    Run dd to make a bit-for-bit copy of the entire drive:

    sudo dd if=/dev/sdd of=memstick.bin bs=1M

    The bs=1M should dramatically speed up the transfer; the default is, IIRC, 512 bytes.

    After this, everything you do uses memstick.bin, not the memory card itself. If you’re paranoid like me, you’ll make a copy of the file before you do anything hazardous. You could even make two copies directly from the memory card and compare them, which might be instructive.

    To find your precious JPG images among the rubble, look for their magic Exif headers:

    strings -t x memstick.bin | grep Exif
      68006 Exif
     258006 Exif
     680006 Exif
     848006 Exif
     a18006 Exif
     bf0006 Exif
     e00006 Exif
     fe8006 Exif
    11e8006 Exif
    1420006 Exif
    ... snippage ...
    

    Because the strings search ignores the (presumably broken) FAT directory structure, you’ll find all the image files on the drive, whether or not they’ve been deleted, partially overwritten, or whatever. This is great for forensics and terrible if you’ve got something to hide. You have been warned.

    The -t x parameter returns the string’s starting offset in hex, which you need to find the actual starting offset of the file: it’s 6 bytes lower! Your mileage may vary, so be prepared to fiddle around a bit.

    You could write a bunch of code to parse the JPG header and extract exactly the number of bytes required, but this is no time for subtle gestures. I just yank out whatever the largest possible image file could be, because image-processing programs only process the valid stuff anyway. So, knowing that this camera produces images in the 2.4 MB range, this extracts the first image:

    dd if=memstick.bin of=image.jpg bs=$((16#1000)) count=1K skip=$((16#68))
    

    The $(( … )) notation evaluates the numeric expression within and the 16#… notation expresses a hexadecimal value.

    Soooo, bs=$((16#1000)) says that the blocksize is 4096 bytes, which you can deduce from the fact that all the Exif headers start 6 bytes from a multiple of 0x1000. Again, your camera may do things differently, but the general notion should get you started.

    If you’re fussy, you’ll note that the headers are actually on multiples of 0x8000 bytes, but using 0x1000 means you can read the high-order digits right off the strings dump. Why make things more complicated than necessary?

    Given a 4K blocksize, count=1K extracts a 4MB chunk: 1024 blocks of 4096 bytes each. That’s larger than the largest possible image file from this camera; adjust it to suit whatever you expect to find. Don’t be stingy, OK?

    The skip=$((16#68)) says to begin extracting data 0x68 blocks into memstick.bin. You read that value directly from the strings output, which is easy enough.

    You could write a tidy Bash script to eat the strings values and spit out the corresponding file chunks. I had few enough images this time to just do it manually, which beats having to debug some code…

    Good luck!