The Smell of Molten Projects in the Morning

Ed Nisley's Blog: Shop notes, electronics, firmware, machinery, 3D printing, laser cuttery, and curiosities. Contents: 100% human thinking, 0% AI slop.

Category: Software

General-purpose computers doing something specific

  • Character-string Histogram

    I’m in the midst of counting glitches on a WWVB receiver, a glitch being any pulse with a duration that isn’t close to 200, 500, or 800 ms. It’s useful to know the pulse width distribution, which is what a histogram does for a living.

    Rather than a graphic histogram, though, I’ve been using a character array (a string!) with one character for each possible pulse duration, measured in units of 20 ms. The value of each character indicates the number of pulses having that duration during the previous minute.

    The counting sequence goes like this:

    • No counts: . (a period)
    • 1 to 9 counts: 1 .. 9
    • 10 to 35 counts: A .. Z
    • 36 to 61 counts: a .. z
    • More than 61 counts: !

    So a minute with no glitches (and a valid WWVB time code frame) looks like this:

    .........4S9............462............322.........

    The three clusters show the three valid pulse widths. The pulse widths from the receiver have an inherent jitter, plus the usual ±1 jitter you get for free when you digitize something, plus (this being RF from half a continent away) whatever the Lords of Cosmic Jest do to the signal. So each width usually occupies two or three cells.

    • The 200 ms binary zero pulses form the cluster on the left: that “S” is equivalent to 10 + 18 = 28 counts. Add in the 4 and 9 on either side and you get 41 binary zero pulses.
    • The middle cluster has the 500 ms binary 1 pulses: 4 + 6 + 2 = 12.
    • Each WWVB time code frame has exactly seven 800 ms frame markers, which form the cluster on the right end: 3+2+2 = 7.

    Add them up: 41 + 12 + 7 = 60. That’s exactly the right number of pulses in a minute. What a pleasant surprise!

    A minute with three out-of-spec glitches looks like this:

    1...1....6NC2............82.......1.....51.........

    And a minute with very high noise that pretty much obliterates the WWVB signal:

    f!!jHC6AB746312.2121..2.1..........................

    Here’s how all that works…

    The histogram counters form a character array that’s also a string. There are 50 20-ms Jiffies in each second (given by the obvious constant), so the histogram has 52 entries: 50 valid counts (0-49), 1 for “more than that”, and 1 for the null byte at the end of the string.

    char    Durations[JIFFIES_PER_SECOND + 2];

    Initialize each counter (character) with the starting value and jam a binary zero at the end:

    memset(Durations,'.',sizeof(Durations)-1);
Durations[sizeof(Durations) - 1] = 0;

    And then tick the appropriate counter as each pulse arrives:

    Index = min(PWM_Width,sizeof(Durations)-2);

switch (Durations[Index]) {
 case '.' :
 Durations[Index] = '1';
 break;
 case '9' :
 Durations[Index] = 'A';
 break;
 case 'Z' :
 Durations[Index] = 'a';
 break;
 case 'z' :
 Durations[Index] = '!';
 break;
 case '!' :
 break;
 default :
 Durations[Index]++;
}

    The switch statement maneuvers the counting sequence through the digits, uppercase and lowercase alphabet, then enforces a stall at the maximum count value of “!”. You can’t just increment each element without some checking, because you do not want unprintable control characters in the string.

    Then you print the histogram as you would any ordinary string. If you’re using an Arduino, as I am, this will suffice:

    Serial.println(Durations);

    All this depends on the ASCII character set’s numerical sequence. Ponder the charts there and all should become clear.

    Here are the histograms from an hour of WWVB reception during the late afternoon of New Year’s Day: watch the noise floor rise up and eat the WWVB signal…

    
.........7OD............36............133..........
 .........BID1...........37.............6.1.........
 .1.......9O91..........262.............43..........
 ........27V2............272............421.........
 .........4Y41...........46............124.........2
 .........AP71...........2811..........123.........2
 .........AN9............731...........1.6..........
 12.111..28IB11.........1623.......2..1.42.....2....
 1412...125Q911..........461.....1......41..........
 1........8Q81..........137...1.........15..........
 .12....119N912.........116..........1.132..........
 1521..1.17O931...1......27...........1..5..........
 .12....12BKB2...........332....1......132.......2..
 25211.1.3AHB5......1...1143........1...14.........3
 45412...4BDA4..12.......25111..1.....1..22.........
 6A44322.26CI53.2.1...1..343...1.......1.11.........
 5D75531113FG511...12..212313.1........1.2..1.......
 1432.1.119GB41..........262.1...11.....14..........
 .5211..115MB4...........441............14.2........
 .4.1....25JB6...........442.........1..231.........
 6B443...27I772..........2322.....1.....1221......2.
 4555411.27HA321........3232......1....114..........
 3232.1.129GI1...1.......4321...1.1.1...111.........
 ...1..1.19O8..1....11...262............24..........
 11......2AN62...........622...........124..........
 111.....2BJ551..1......1522........3..141........21
 1421....29N611.......1.136............1231........2
 1.12.....AIB3........1..38...1........312..........
 7F42..1.28FE341.1.1.....351........1...12...1.....2
 .23......8GG2....11....2423............33..........
 112..13.1AK931.....2....142....2.11.1..3.1........2
 423111..44JD4.2..1.....1433........11111...........
 1413..1.3CL812..11......242..1....11...21.........2
 474112.34AI84.2...3.1...16.2.....1...1.11..........
 8LHC734657D857411..11...131......1..1.1...........2
 JuSFDCCDDCD662412...1..............................
 VmG376788AC8671.21.1..3.......1....................
 IlCB4576CFE532131.11.1..1.1........................
 KxN798EB98A7422...2.1...1..........................
 AYL96853CF7742121311.13..21....1...................
 8TDEB6649A7952..32.41..124.....1.....1............2
 DeO78638C9GA6142.15111..........1........2.....2...
 AcPC83426BD4823.122..11132.1.......................
 7SF31121AHFB73..2.111...321..1.1...................
 7F3221414GG5411.111121.133...1.1.1.................
 6RD9669647F7361.31.2..1212...1.....................
 6L71.2444ALB21...221.212321..2...1...1.............
 4LHFD7638C9B12.31341.22113...2..........1..........
 264321112EFA5.112.22.2.14111..1.....1...1.........2
 235511445BEC3.12..1131.131..1.1........11..........
 7UC741635AAC442.1..112.1..11311.......1.1.........2
 EVLCAB3598E752252133.111.1.1..........1...........2
 JnTE9913A59A452331.13..1222.............1..........
 f!!jHC6AB746312.2121..2.1..........................
 !!!iMB772.4532..1.1.1.1..1.........................

  • WWVB Time Code Format Diagram

    This is by-and-large the same diagram of the WWVB Time Code Format that you’ll find there, but with:

    • the PR frame reference pulse identified
    • the last four bits clearly labeled
    • some verbiage chopped out
    WWVB Time Code Format - with PR marker
    WWVB Time Code Format – with PR marker

    Memo to Self: Remember that…

    • The time code applies to the minute that’s currently happening, so …
    • The PR pulse happens before the rest of the time code arrives
    • You must know the end-of-month date to apply the leap second bit, but EOM isn’t part of the data stream

  • Extracting Text from PDF Documents

    Mary had to extract an extensive table from a PDF (think financial statements) and found that a simple cut-and-paste failed with mysterious symptoms. I’ve had that happen, too, as PDF documents sometimes have a complete disconnect between the rendered page and the original text; sometimes you can’t even select the text block you want without copying huge chunks of the surrounding document or pasting meaningless junk.

    Easy solution: feed the PDF into pdftotext and extract the table from the ensuing flat text file.

    It’s a command-line thing:

    
pdftotext -layout whatever.pdf

    That produces whatever.txt with the ASCII characters bearing more-or-less the same spatial arrangement as the original PDF, minus all the font and graphic frippery. It tends to insert a ton of blanks in an attempt to make the formatting come out right, which may not be quite what you want.

    Omitting the -layout option gives you something vaguely resembling the PDF, although precisely arranged tables tend to fare poorly.

    If you have a bazillion-page PDF document and need the text from just a page or two, feed it into the pdftk brush chipper, extract the appropriate pages, and then run those files through pdftotext. You can probably get similar results using just pdftk, but pdftotext seems to work better on the files I’ve had to deal with.

    This is a GNU/Linux thing; the programs are likely part of your favorite distribution; follow the links if not. If you’re still using Windows, maybe they’ll work for you, but maybe it’d be easier to just go buy something similar.

  • WWVB Reception Quality

    Here on the East Coast of the US, WWVB reception is iffy during the day, due to low signal strength and high ambient noise. Actual data seems hard to come by, so here’s a small contribution.

    This is a plot of the number of glitches per minute, where a glitch is any pulse that’s not within ±60 ms of the expected pulse durations (200, 500, and 800 ms), for a 24-hour period starting at UTC 0257 on 25 Dec 2009 (9:57 EST Christmas Eve 2009). There are 1448 data points, each representing the glitches during the previous minute; each minute starts within 2 seconds of the WWVB on-time frame marker.

    Here’s the raw data, log-scaled on the Y axis to cover the dynamic range. Log scaling can’t handle 0-valued points, so I forced counts of 0 to 0.1 to make them visible.

    Glitchiness - raw - 24 hrs - 2009-12-24
    Glitchiness – raw – 24 hrs – 2009-12-24

    Here’s the same data, Bezier smoothed to make the trends more obvious;  all the points below 1.0 are approximations of a trend toward counts of 0.

    Glitchiness - smoothed - 24 hrs - 2009-12-24
    Glitchiness – smoothed – 24 hrs – 2009-12-24

    Even better, splines show the glitch-free minutes without forcing the data points.

    Glitchiness - splines - 24 hrs - 2009-12-24
    Glitchiness – splines – 24 hrs – 2009-12-24

    My firmware requires four successive glitch-free minutes of reception (plus some additional verification) before synching its local time to WWVB, so it’s exceedingly fussy. Despite that, it still synched 17 times during those 24 hours. The longest free-running time between synchs was 6.8 hours.

    Note that there are 17 downward peaks below 1.0 in that last graph.

    Winter is, of course, the time of best ground-wave propagation from WWVB, so this is about as good as it’s ever going to get.

    Memo to Self: useful Bash and Gnuplot commands…

    
grep Glitch WWVB_2009-12-24a.log | cut -d H -f 1 > Glitches.txt

set logscale y

set samples 250

plot 'Glitches.txt' using ($2<1?0.1:$2) with points lt 3 pt 2

plot 'Glitches.txt' using 2 smooth csplines with linespoints lt 3 pt 0
    log-scaled on the Y axis to cover the dynamic range. Log scaling can’t handle 0-valued points, so I forced them to 0.1;

  • HP54602 Oscilloscope Trace Conversion Tweakage

    The script (writeups there and there) I use to convert the HPGL screen dumps from my HP54602 into PNG images produced a transparent background. I put the files into an OpenOffice mockup of my Circuit Cellar columns and the background turns white, so I figured it worked OK.

    Turns out that the workflow at Circuit Cellar Galactic HQ turns the background black. A bit of digging showed that the ImageMagick convert program produced an alpha channel that selected only the traces and left everything else unselected. Why that produces white here and black there is a mystery, but there’s no point in putting up with such nonsense.

    Another wrestling match produced this revision (the two changed lines are highlighted), which has no alpha channel and a white background. That ought to simplify things: an image shouldn’t depend on where it’s dropped to look right.

    #!/usr/bin/kermit +
# Fetches screen shot from HP54602B oscilloscope
# Presumes it's set up for plotter output...
# Converts HPGL to PNG image

set modem none
set line /dev/ttyUSB0
set speed 19200
set flow rts/cts
set carrier-watch off

# Make sure we have a param
if not defined \%1 ask \%1 {File name? }

set input echo off
set input buffer-length 200000

# Wait for PRINT button to send the plot
echo Set HP54602B for HP Plotter, FACTORS ON, 19200, DTR
echo Press PRINT SCREEN button on HP54602B...

log session "\%1.hgl"

# Factors On
input 480 \x03

close session
close

echo Converting HPGL in
echo --\%1.hgl
echo to PNG in
echo --\%1.png

# Factors Off
#run hp2xx -q -m png -a 1.762 -h 91 -c 14 "\%1.hgl"
#run mogrify -density 300 -resize 200% "\%1.png"

# Factors On
run sed '/lb/!d' "\%1.hgl" > "\%1-1.hgl"
run hp2xx -q -m eps -r 270 -a 0.447 -c 14 -f "\%1.eps" "\%1-1.hgl"
run rm "\%1-1.hgl"
run convert "\%1.eps" -alpha off -resize 675x452 "\%1.png"

echo Finished!

exit 0

  • TLC5916 Configuration Code Setting

    TLC5916 Writing Config Code
    TLC5916 Writing Config Code

    The TLC5916 data sheet clearly shows that you write the configuration code (which controls the LED current) by shifting seven bits in, then raising LE during the 8th SCK pulse while simultaneously shifting the 8th bit.

    That makes no sense whatsoever: you couldn’t use standard SPI hardware in a chained configuration, because you’d have to blip LE while shifting.

    In fact, the chip doesn’t work that way. You set the config code in Special Mode just like you set the LED driver bits in Normal Mode: shift ’em all in, then blip LE to latch ’em into the parallel holding register.

    Here’s the code to make it happen…

    DisableSPI();                               // manual SPI control

 digitalWrite(PIN_DISABLE_DISPLAY,HIGH);    // initial condition
 digitalWrite(PIN_LATCH_DO,LOW);

 PulsePin(PIN_SCK);                         // 1
 digitalWrite(PIN_DISABLE_DISPLAY,LOW);
 PulsePin(PIN_SCK);                         // 2
 digitalWrite(PIN_DISABLE_DISPLAY,HIGH);
 PulsePin(PIN_SCK);                         // 3
 digitalWrite(PIN_LATCH_DO,HIGH);           //   sets Special Mode
 PulsePin(PIN_SCK);                         // 4
 digitalWrite(PIN_LATCH_DO,LOW);
 PulsePin(PIN_SCK);                         // 5

//-- Send brightness level

 EnableSPI();                               // turn on SPI hardware

 SendRecSPI(Brightness);
 SendRecSPI(Brightness);
 SendRecSPI(Brightness);
 SendRecSPI(Brightness);
 SendRecSPI(Brightness);

 PulsePin(PIN_LATCH_DO);                    // latch new shift reg contents into drivers

//-- put LED drivers back in Normal Mode

 DisableSPI();

 digitalWrite(PIN_DISABLE_DISPLAY,HIGH);     // initial condition
 digitalWrite(PIN_LATCH_DO,LOW);

 PulsePin(PIN_SCK);                          // 1
 digitalWrite(PIN_DISABLE_DISPLAY,LOW);
 PulsePin(PIN_SCK);                          // 2
 digitalWrite(PIN_DISABLE_DISPLAY,HIGH);
 PulsePin(PIN_SCK);                          // 3
 digitalWrite(PIN_LATCH_DO,LOW);             //   sets Normal Mode
 PulsePin(PIN_SCK);                          // 4
 digitalWrite(PIN_LATCH_DO,LOW);
 PulsePin(PIN_SCK);                          // 5

 digitalWrite(PIN_DISABLE_DISPLAY,LOW);      // turn the LEDs on again

    The SendRecSPI() function does exactly what you’d expect:

    byte SendRecSPI(byte Dbyte) {                // send one byte, get another in exchange

 SPDR = Dbyte;                      // assume it's OK to send a new byte

 while (! (SPSR & (1 << SPIF))) {   // wait for shift to finish
  continue;
 }

 return SPDR;                       // SPIF will be cleared
}

    I don’t know. Maybe the chip also works the way they show in the datasheet, but I doubt it’s worth finding out.

  • TLC5916 LED Driver Current Monotonicity Hack

    TLC5916 Current Gain vs Config Code
    TLC5916 Current Gain vs Config Code

    I’m using Texas Instruments TLC5916 constant-current LED driver chips for my my friend’s Totally Featureless Clock. An 8-bit configuration value sets the output current, with the external resistor defining the maximum value as described there.

    The problem is that the current-versus-config-value curve has a non-monotonic discontinuity in the middle, where the Current Multiplier bit switches from 0 to 1. I don’t know in which alternate universe this design decision made sense, but right here and now…

    It. Does. Not.

    Why it’s a problem: the LED brightness tracks room illumination as seen by a CdS photoresistor, averaged over maybe half a minute. The brightness changes very slowly, so the jump when it goes from 0x7f to 0x80 is really eyecatching. At least to me, anyway.

    Eyeballometric measurement of the curve shows the current at 0x80 pretty much matches the current at 0x60, soooo let’s just shove the second quarter of the curve (between 0x40 and 0x7f) downward until it meets the high-current value at 0x80.

    This code does the trick:

    if ((Brightness >= 0x40) && (Brightness <= 0x7f)) {
 Brightness = 0x40 + ((Brightness & 0x3f) >> 1);
}

    Basically, that maps 0x7f to 0x5f. The output current for 0x5f is pretty close to the current for 0x80, making the step pretty much a non-issue.

    You could, if you were fussy enough, work out the actual current mapping values from the data sheet equations and make the ends match up perfectly.