Ed Nisley's Blog: Shop notes, electronics, firmware, machinery, 3D printing, laser cuttery, and curiosities. Contents: 100% human thinking, 0% AI slop.
Category: Software
General-purpose computers doing something specific
I’m doing some work with a one-off ROM reader & EPROM programmer, so it’s once again time to mess around with Intel HEX files, raw binary images, and the like.
The key routine (which runs on an Arduino Decimila) to dump a ROM in HEX format goes like this, with all the constants & variables & functions doing the obvious things:
void DumpHC641(void) {
word Address,Offset;
byte DataRd,Checksum;
for (Address = 0; Address < ROM_SIZE; Address += IHEX_BYTES) {
sprintf(PrintBuffer,":%02X%04X00",IHEX_BYTES,(word)Address); // emit line header
Serial.print(PrintBuffer);
Checksum = IHEX_BYTES + lowByte(Address) + highByte(Address) + 0x00; // record type 0x00
for (Offset = 0; Offset < IHEX_BYTES; ++Offset) {
digitalWrite(PIN_HEARTBEAT,HIGH);
DataRd = ReadHC641(Address + Offset);
digitalWrite(PIN_HEARTBEAT,LOW);
Checksum += DataRd;
sprintf(PrintBuffer,"%02X",DataRd); // data byte
Serial.print(PrintBuffer);
}
Checksum = -Checksum; // two's complement
sprintf(PrintBuffer,"%02X",Checksum);
Serial.println(PrintBuffer);
}
Serial.println(":00000001FF"); // emit end-of-file line
}
So getting an Intel HEX file is just a matter of capturing the serial output, whacking off any debris on either side of the main event, and saving it.
The srec_cat program handles conversions among a myriad formats, most of which I can’t even pronounce. The few I use go a little something like this:
It’s sometimes handy to apply srec_cat to a group of similarly suffixed files, in which case some Bash string chopping comes in handy. For example, to convert some hex files into binary:
for f in 27HC641*hex ; do echo ${f%%hex} ; srec_cat "$f" -intel -o "${f%%hex}"bin -binary ; done
Good old diff works fine on text files, but in this case it’s better to see which bytes have changed, rather than which lines (which don’t apply in the context of a binary file). The vbindiff program looks great on a portrait-mode display.
I don’t do much binary editing, but tweak serves my simple needs. Confusingly, members of this class of program are called “hex editors”, but they really work on binary files.
There’s also diff3, for those rare cases where you must mutually compare three text files. Makes my head spin every time…
All those programs are likely packages in your favorite Linux distro.
Many interesting projects require more digital output bits than the Arduino hardware can support. You then use 74HC595 serial-in/parallel-out chips and that tutorial pretty well explains how it works. The shiftOut() library function squirts a byte out through an arbitrary pin, leading with either the high or low bit.
Software SPI: Clock and Data
Just drop one byte into shiftOut() for each ‘595 lined up on your board. Remember to latch the bits (LOW-to-HIGH on RCK @ pin 12 of the ‘595) and enable the output drivers (LOW on -G @ pin 13, similarly) when you’re done sending everything. You can have separate latches-and-enables for each ‘595 if that suits your needs, although then you once again run out of Arduino bits pretty quickly. It’s entirely possible to devote a ‘595 to latches-and-enables for the rest of the chain, but that gets weird in short order.
The scope shot shows that shiftOut() ticks along at 15 µs per bit (clock in the upper trace, data in the lower trace). For back-of-the-envelope purposes, call it 8 kB/s, which is probably less than you expected. If you have a string of 5 external bytes, as I did on a recent project, that’s only 1600 updates / second. It was part of a memory board reader & EPROM programmer: reading an 8 kB ROM chip requires two shift-register runs (one to set the address & data, one to read in the chip output), so the overall rate was on the order of 10 seconds per pass and much worse for programming. You can optimize the number of bits by not shifting out all the bytes, but that’s the general idea.
Because ‘595 chips are output-only, in order to get 8 bits of data into the Arduino board, add a 74HC166 parallel-in/serial-out chip to the string. Alas, shiftOut() doesn’t know about input bits, so you’re on your own.
Hardware SPI: Clock and Data
If you’re going to have to write some code to get input bits anyway, you may as well use the ATmega168 (and its ilk) hardware SPI as it was intended to be used: for high-speed synchronous serial I/O. This scope shot shows the SPI clock (in the top trace again) ticking along at 1 µs per bit, which is 1/16 the Diecimila’s oscillator frequency. You can pick any power of two between 1/2 and 1/128; I used 1/16 because it’s fast enough to make the rest of the software the limiting factor, while slow enough to not require much attention to layout & so forth.
Start by Reading The Fine Manual section about the ATmega168’s SPI hardware, starting at page 162.
The pin definitions, being lashed to internal hardware, are not optional. Note that SCK is also the standard Arduino LED, which won’t be a problem unless you need a tremendous amount of drive for a zillion ‘595s. I stuck an additional LED on Arduino digital pin 2.
#define PIN_HEARTBEAT 2 // added LED
#define PIN_SCK 13 // SPI clock (also Arduino LED!)
#define PIN_MISO 12 // SPI data input
#define PIN_MOSI 11 // SPI data output
Initial hardware setup goes in the usual setup() function:
pinMode(PIN_SCK,OUTPUT); // set up for "manual" SPI directions
digitalWrite(PIN_SCK,LOW);
pinMode(PIN_MOSI,OUTPUT);
digitalWrite(PIN_MOSI,LOW);
pinMode(PIN_MISO,INPUT); // configure inputs
digitalWrite(PIN_MISO,HIGH);
SPCR = B01110001; // Auto SPI: no int, enable, LSB first, master, + edge, leading, f/16
SPSR = B00000000; // not double data rate
Basically, the “manual” setup allows you to wiggle the bits by hand with the hardware SPI control disabled.
Arduino Hardware SPI Schematic
Here’s a chunk of the schematic so you can see how the bits rattle around. You’ll surely want to click it to get the details…
I put the data in a structure that matches the shift register layout, with the first byte (Controls) connected to the ATmega’s MOSI pin and the last byte (DataIn) connected to MISO. The SCK pin drives all of the serial clock pins on the ‘595 and ‘166 chips in parallel. Your structure will certainly be different; this was intended to suck data from a Tek 492 Spectrum Analyzer memory board.
typedef struct { // external hardware shift register layout
byte Controls; // assorted control bits
word Address; // address value
byte DataOut; // output to external devices
byte DataIn; // input from external devices
} SHIFTREG;
SHIFTREG Outbound; // bits to be shifted out
SHIFTREG Inbound; // bits as shifted back in
The functions that make it happen are straightforward:
void TogglePin(char bitpin) {
digitalWrite(bitpin,!digitalRead(bitpin));
}
void PulsePin(char bitpin) {
TogglePin(bitpin);
TogglePin(bitpin);
}
void EnableSPI(void) {
SPCR |= 1 << SPE;
}
void DisableSPI(void) {
SPCR &= ~(1 << SPE);
}
void WaitSPIF(void) {
while (! (SPSR & (1 << SPIF))) {
// TogglePin(PIN_HEARTBEAT); // use these for debugging!
// TogglePin(PIN_HEARTBEAT);
continue;
}
}
byte SendRecSPI(byte Dbyte) { // send one byte, get another in exchange
SPDR = Dbyte;
WaitSPIF();
return SPDR; // SPIF will be cleared
}
void CaptureDataIn(void) { // does not run the shift register!
digitalWrite(PIN_ENABLE_SHIFT_DI,LOW); // allow DI bit capture
PulsePin(PIN_SCK); // latch parallel DI inputs
digitalWrite(PIN_ENABLE_SHIFT_DI,HIGH); // allow DI bit shifting
}
void RunShiftRegister(void) {
EnableSPI(); // turn on the SPI hardware
Inbound.DataIn = SendRecSPI(Outbound.DataIn);
Inbound.DataOut = SendRecSPI(Outbound.DataOut);
Inbound.Address = SendRecSPI(lowByte(Outbound.Address));
Inbound.Address |= ((word) SendRecSPI(highByte(Outbound.Address))) << 8;
Inbound.Controls = SendRecSPI(Outbound.Controls);
PulsePin(PIN_LATCH_DO); // make new shift reg contents visible
PulsePin(PIN_LATCH_ADDRESS);
PulsePin(PIN_LATCH_CONTROLS);
DisableSPI(); // return to manual control
}
Actually using the thing is also straightforward. Basically, you put the data-to-be-sent in the Outbound variables and call RunShiftRegister(), which drops output bytes into SPDR and yanks incoming bytes out, then stuffing them in the Inbound variables. I have separate latch controls for the Controls, Address, and Data chips, although I don’t use them separately here.
You must wiggle the parallel latch enable line on the 74HC166 chip before shifting to capture the data, as shown in CaptureDataIn(). That chip also requires a separate pulse on its serial clock line to latch the data, which you do manually with the hardware SPI disabled. If you’re paying attention, you’ll wonder if that clock pulse also screws up the data in the rest of the chips: yes, it does. If this is a problem, you must add some external clock-gating circuitry, disable the ‘595s, or pick a different input shift register chip; it wasn’t a problem for what I was doing.
Here’s a function that reads data from a RAM chip on the Tek memory board, so it must write the address and read the RAM chip’s output. The PIN_DISABLE_DO bit controls the output buffers on the ‘595 that drives the RAM’s data pins; they must be disabled to read data back from the RAM. Don’t worry about the other undefined bits & suchlike; just assume everything does what the comments would have you believe.
byte ReadRAM(word Address) {
digitalWrite(PIN_DISABLE_DO,HIGH); // turn off data latch output
digitalWrite(PIN_BUS_READ,HIGH); // allow RAM read access
Outbound.Controls |= CB_BUS_CLKPH2_MASK; // set up RAM -CS gate
Outbound.Address = Address;
Outbound.DataOut = 0x55; // should not be visible
RunShiftRegister();
digitalWrite(PIN_BUS_N_SYSRAM,LOW); // activate RAM -CS
CaptureDataIn(); // latch RAM data
digitalWrite(PIN_BUS_N_SYSRAM,HIGH); // ... and turn -CS off
Outbound.Controls &= ~CB_BUS_CLKPH2_MASK; // disable -CS gate
RunShiftRegister(); // tell the board and get data
return Inbound.DataIn;
}
Hardware SPI – Detail of clock and data timing
Here’s a detailed shot of the outbound bit timing. Notice that the upward clock transitions shift bits into the ‘595 and ‘166 chips, while the SPI output data changes on the downward transitions. You can tweak that to match your hardware if you’re using different shift register chips, by messing with the SPCR settings.
Bottom line: using the ATmega168 hardware SPI provided a factor-of-15 speedup and serial digital input, too.
Gnuplot can do curve fitting (of all kinds) and parks the coefficients in variables. In general, you’d like to display those values on the final plot for later reference…
The trick is using the sprintf() function, which behaves largely like the C version, to insert the variable into a formatted string for use in the label command.
I drive Gnuplot with shell scripts, which simplifies introducing parameters & suchlike. That’s conspicuous by its absence here, but when you need it, you need it bad.
The script to generate that plot looks like this, with some key points in the highlighted lines:
#!/bin/sh
export GDFONTPATH="/usr/share/fonts/truetype/msttcorefonts/"
gnuplot << EOF
set term png font "arialbd.ttf" 18 size 950,600
set output "Calibration Curve - Full.png"
set title "Calibration Curve - Full"
set key noautotitles
unset mouse
set bmargin 4
set grid xtics ytics
set xlabel "10^5/ADC"
set format x "%3.0f"
set ylabel "Resistance - Ohm"
set format y "%3.0f"
set yrange [0:100]
set datafile separator "\t"
f(x) = m*x + c
fit f(x) "Measurements/Calibration.csv" using 3:1 via m,c
set label 1 sprintf("m = %3.4f",m) at 510,75 font "arialbd,18"
set label 2 sprintf("c = %3.4f",c) at 510,70 font "arialbd,18"
plot \
"Measurements/Calibration.csv" \
using 3:1 with linespoints lt 3 lw 3 pt 3 , \
f(x) lt 4 lw 2
EOF
The dataset for that plot is tucked into the obvious file and looks like this, with tabs between the columns:
For those of you still using Windows, here’s a sobering look at why you shouldn’t: an analysis of the Torpig botnet by an academic group that managed to take over its command & control structure for a few days.
The report is tech-heavy, but well worth the effort to plow through.
Here are some of the high points…
Why do the bad guys do this? It’s all about the money, honey:
In ten days, Torpig obtained the credentials of 8,310 accounts at 410 different institutions.
… we extracted 1,660 unique credit and debit card numbers from our
collected data.
Does an antivirus program help?
Torpig has been distributed to its victims as part of Mebroot. Mebroot is a rootkit that takes control of a machine by replacing the system’s Master Boot Record (MBR). This allows Mebroot to be executed at boot time, before the operating system is loaded, and to remain undetected by most anti-virus tools
In these attacks, web pages on legitimate but vulnerable web sites are modified with the inclusion of HTML tags that cause the victim’s browser to request JavaScript code from a[nother] web site under control of the attackers. This JavaScript code launches a number of exploits against the browser or some of its components, such as ActiveX controls and plugins. If any exploit is successful, an executable is downloaded from the drive-by-download server to the victim machine, and it is executed.
What happens next?
Mebroot injects these modules […] into a number of applications. These applications include the Service Control Manager (services.exe), the file manager, and 29 other popular applications, such as web browsers (e.g., Internet Explorer, Firefox, Opera), FTP clients (Leech-FTP, CuteFTP), email clients (e.g., Thunderbird, Outlook, Eudora), instant messengers (e.g., Skype, ICQ), and system programs (e.g., the command line interpreter cmd.exe). After the injection, Torpig can inspect all the data handled by these programs and identify and store interesting pieces of information, such as credentials for online accounts and stored passwords.
If you think hiding behind a firewall router will save you, you’re wrong:
By looking at the IP addresses in the Torpig headers we are able to determine that 144,236 (78.9%) of the infected machines were behind a NAT, VPN, proxy, or firewall.
If you think you’ve got a secure password, you’re wrong:
Torpig bots stole 297,962 unique credentials (i.e., username and password pairs), sent by 52,540 different Torpig-infected machines over the ten days we controlled the botnet
If you think a separate password manager will save you, you’re wrong.
It is also interesting to observe that 38% of the credentials stolen by Torpig were obtained from the password manager of browsers, rather than by intercepting an actual login session.
Remember, the virus / worm / Trojan / botnet attacks you read about all the time only affect Windows machines. Linux isn’t invulnerable, but it’s certainly safer right now. If you’re running Windows, it’s only a matter of time until your PC is not your own, no matter how smart you think you are.
If you have one or two must-gotta-use Windows programs, set up a dedicated Token Windows Box and use it only for those programs. Network it (behind a firewall) if you like, but don’t do any email / Web browsing / messaging / VOIP on it. Just Say No!
For everything else, run some version of Linux. It’ll do what you need to get done with less hassle and far less risk. It’s free for the download, free for the installation, and includes all the functions you’re used to paying money for. Just Do It!
If you think using Linux is too much of a hassle, imagine what putting your finances back together will be like. Remember, the bad guys will steal everything you’ve ever put on your PC, destroy your identity, and never get caught.
Having determined that the existing Arduino LiquidCrystal library routine wouldn’t work for the Optrex DMC-16117 character LCD in my parts heap, I decided to modify it to meet the data and timing requirements mentioned in the datasheet. This is sufficiently slow and old that it should work for contemporary displays built around the Hitachi HD44780 and its ilk, but I certainly haven’t tested it!
The straight dope on building an Arduino library from scratch is there, but there’s no need to work from First Principles here.
Start by copying the library files (this FLOSS stuff is wonderful that way), renaming them, and changing all the LiquidCrystal strings to LCD_Optrex:
cd /opt/arduino/hardware/libraries/
cp -a LiquidCrystal LCD_Optrex
cd LCD_Optrex
rm LiquidCrystal.o
rename 's/LiquidCrystal/LCD_Optrex/' LiquidCrystal.*
for f in LCD* k*; do sed -i 's/LiquidCrystal/LCD_Optrex/g' $f; done
cd examples
for f in * ; do sed -i 's/LiquidCrystal/LCD_Optrex/g' $f/$f.pde
cd -
You could do that by hand with an editor if you prefer.
Depending on how you’ve installed the Arduino files, you may need a sudo to make that work. Better, perhaps, to tweak the permissions for (at least) the LCD_Optrex directory & files therein to grant yourself write access.
I created a sendraw4() function to send a single 4-bit nibble during the startup sequence, so add that to the private section of LCD_Optrex.h:
The new code is in LCD_Optrex is shamelessly adapted from the existing send() function, minus the mode selection and 8-bit stuff:
void LCD_Optrex::sendraw4(uint8_t value) {
digitalWrite(_rs_pin, LOW);
digitalWrite(_rw_pin, LOW);
for (int i = 0; i < 4; i++) {
digitalWrite(_data_pins[i], (value >> (i + 4)) & 0x01);
}
digitalWrite(_enable_pin, HIGH);
digitalWrite(_enable_pin, LOW);
}
If I were doing this from scratch, I’d use d7 through d4 rather than d3 through d0 to match the datasheet, but that’s a stylin’ thing.
Replace the existing LCD panel setup code with an exact mapping of the datasheet’s procedure. For the 4-bit setup, it goes a little something like this:
delayMicroseconds(16000); // mandatory delay for Vcc stabilization
sendraw4(0x30); // set 8-bit mode (yes, it's true!)
delayMicroseconds(5000); // mandatory delay
sendraw4(0x30);
delayMicroseconds(200);
sendraw4(0x30);
delayMicroseconds(40); // command delay
sendraw4(0x20); // finally set 4-bit mode
delayMicroseconds(40); // command delay
command(0x28); // 4-bit, 2-line, 5x7 char set
command(0x08); // display off
command(0x01); // clear display
delayMicroseconds(16000);
command(0x06); // increment, no shift
command(0x0c); // display on, no cursor, no blink
It seems you cannot use the delay() function in the constructor, as interrupts and suchlike aren’t active. The delayMicroseconds() function disables & enables interrupts; I don’t know if that is a Bad Thing or not.
The 8-bit initialization code, which I haven’t tested, doesn’t need the sendraw4() function, but does need the same alterations. Apart from enabling 4-bit mode, of course.
Various commands have different timing requirements, as shown on page 39 of the DMC16117 datasheet. Add a delayMicroseconds(16000); to the clear() and home() functions, then add delayMicroseconds(40); to the send() function, like this:
void LCD_Optrex::clear()
{
command(0x01); // clear display, set cursor position to zero
delayMicroseconds(16000);
}
Optrex DMC16117 Instruction Timing
With all that in place, fire up the Arduino IDE and compile one of the example programs. That will build the LCD_Optrex.o file, too. If you have such a display, either wire it up as indicated or change the example code to match your connections.
What should happen is that the LCD should initialize correctly under all conditions… how’s that for anticlimactic?
Here’s an OpenOffice document with LCD_Optrex.h, LCD_Optrex.cpp, and examples.txt all in one lump: LCD_Optrex Library Files.odt. Save each section as a separate flat-ASCII text file with the appropriate name in the right spot and you’re in business. I’d present a ZIP file, but WordPress isn’t up to that.
Memo to Self: A function to write a 16-character string to the stupid 16-character DMC16117 which has a single row that’s addressed as two 8-character lines would be nice. That requires keeping track of the current cursor position, which could be tricky. Maybe I should just scrap those suckers out?
I recently attached an ancient Optrex DM16117 LCD to an Arduino and discovered that the standard LiquidCrystal library routine wouldn’t initialize it properly. After turning on the power, the display would be blank. Hitting the Reset button did the trick, but that’s obviously not the right outcome.
It turns out that initializing one of these widgets is trivially easy after you realize that the data sheet is required reading. If you do everything exactly right, then it works; get one step wrong, then the display might work most of the time, sorta-kinda, but most likely it won’t work, period.
The catch is that there’s no such thing as a generic datasheet: what you must do depends on which version of the HD44780 controller lives on the specific LCD board in your hands and what oscillator frequency it’s using. The LiquidCrystal library seems to be written for a much newer and much faster version of the HD44780 than the one on my board, but, even so, the code may not be following all the rules.
Optrex DMC16117 Initialization Sequence
To begin…
Fetch the Optrex DMC16117 datasheet, which includes the HD44780 timings for that family of LCD modules. There’s also a datasheet for just the Optrex LCD module itself, which isn’t quite what you want. You could get a bare Hitachi HD44780 datasheet, too, but it won’t have the timings you need.
Pages 32 and 33 of the DMC16117 datasheet present the 8-bit and 4-bit initialization sequences. Given that no sane engineer uses the 8-bit interface, here’s the details of the 4-bit lashup.
Two key points:
The first four transfers are not standard command sequences
The delays between transfers are not negotiable
The starting assumption is that the LCD has not gone through the usual power-up initialization, perhaps because the supply voltage hasn’t risen at the proper rate. You could drive the LCD power directly from a microcontroller pin for a nice clean edge, but most designs really don’t have any pins to spare for that sort of nonsense: code is always cheaper than hardware (if you ignore non-recurring costs, that is, as many beancounters do).
The Arduino LiquidCrystal library routine initialization sequence (in /opt/arduino/hardware/libraries/LiquidCrystal/LiquidCrystal.cpp) looks like this:
command(0x28); // function set: 4 bits, 1 line, 5x8 dots
command(0x0C); // display control: turn display on, cursor off, no blinking
command(0x06); // entry mode set: increment automatically, display shift, right shift
clear();
The four-bit version of the command() function sends both nibbles of its parameter, high followed by low, which simply isn’t correct for the first few values the DMC16117 expects. Worse, the timing doesn’t follow the guidelines; there’s no delay at all between any of the outgoing values. Again, this is most likely due to the fact that LiquidCrystal was written for a newer version of the HD44780 chip.
After a bit of fiddling around, I decided that the only solution was to create a new library routine based on LiquidCrystal with the proper delays and commands: LCD_Optrex. It might not work for newer LCDs, but at least it’ll play with what I have in my parts heap.
Every now and again the Arduino IDE spits out an error message along the lines of “couldn’t determine program size” or simply fails to compile with no error message at all. The former is evidently harmless, but the latter can be truly annoying.
The cause is described for Macs there as a race condition in the IDE on multi-core processors, with a patch that either partially fixes the problem or pushes it to a less-likely part of the code. That’s true on my system, as the error still occurs occasionally.
How you apply it to Xubuntu 8.10: unzip the file to get Sizer.class, then copy that file to /usr/lib/jvm/java-6-sun-1.6.0.10/jre/lib/. That won’t be the right place for a different Xubuntu or different Java, so use locate rt.jar and plunk it into that directory.
A less dramatic change seems to be setting build.verbose=true and upload.verbose=true in ~/.arduino/preferences.txt.
This is evidently an error of long standing, as it’s been discussed since about Arduino 11. I’m currently at 15 and it seems that patch will be in the next version of the IDE.