Category: Science

The patio at the rear of our house is a six inch concrete slab with a five foot “crawlspace” below it, with a HOBO datalogger pendant in the dirt near the pole at the middle of the I beam supporting the slab. The pendant is very close to the surface, so there’s a big diurnal temperature variation, but it still gives a reasonable picture of seasonal change.

A configuration setting in Hoboware determines whether it stores / exports dates with years having two digits or four digits. As you might expect, over the course of five years the dates have both formats, but there’s always a blank separating the date and the time:

grep "/12 " /tmp/data.csv | head -1
1,11/09/12 08:45:00 ,49.050,,,,
grep "/2012 " /tmp/data.csv | head -1
2235,01/01/2012 00:00:00,44.560,0.0,,,,

This burst of sed regex line noise normalizes all years to four digits:

sed 's/\/\([0-9][0-9]\) /\/20\1 /' whatever.csv

The parenthesized subexpression matches the digits of a two digit year preceding a blank, then the \1 plugs it into the right spot in the output. This suffers from the usual failure when the century rolls over, but frankly, my dear readers, I don’t give a damn. The backslashes escape forward slashes and parentheses, in addition to making the regex pretty much write-only.

The plot shows the expected annual variation:

The periodic upward spikes happen when I carry the logger to the Token Windows Laptop and read it out; the air temperature upstairs is always warmer than the dirt under the patio.

The Bash and gnuplot script that produced the graph:

#!/bin/sh
#-- overhead
export GDFONTPATH="/usr/share/fonts/truetype/"
base="${1%.*}"
echo Base name: ${base}
ofile=${base}.png
tfile=$(tempfile)
echo Input file: $1
echo Temporary file: ${tfile}
echo Output file: ${ofile}
#-- prepare csv Hobo logger file
sed 's/^\"/#&/' "$1" | sed 's/^.*Logged/#&/' | sed 's/ ,/,/' | sed 's/\/\([0-9][0-9]\) /\/20\1 /' > ${tfile}
#-- do it
gnuplot << EOF
#set term x11
set term png font "arialbd.ttf" 18 size 950,600
set output "${ofile}"
set title "${base}"
set key noautotitles
unset mouse
set bmargin 4
set grid xtics ytics
set timefmt "%m/%d/%Y %H:%M:%S"
set xdata time
set xlabel "Date"
set format x "%Y-%m"
#set xrange [1.8:2.2]
set xtics font "arial,12"
#set mxtics 2
#set logscale y
#set ytics nomirror autofreq
set ylabel "Temperature - F"
#set format y "%4.0f"
#set yrange [30:90]
#set mytics 2
#set y2label "right side variable"
#set y2tics nomirror autofreq 2
#set format y2 "%3.0f"
#set y2range [0:200]
#set y2tics 32
#set rmargin 9
set datafile separator ","
#set label 1 "label text" at 2.100,110 right font "arialbd,18"
#set arrow from 2.100,110 to 2.105,103 lt 1 lw 2 lc 0
plot	\
    "${tfile}" using 2:3 with lines lt 3
EOF

Starting in late 2006, I’ve have several Hobo dataloggers recording the temperature / humidity / light at various locations, under the principle that if you observe something long enough, it turns into science. Regrettably, logging the data is one thing, actually processing it into usable information is entirely another; there’s never a good time for the latter. Perhaps if I break it down into monthly chunks, I can actually make some progress on getting it done.

The first problem is that the Hobo dataloggers lack a convenient user interface: the only way to extract data is through the Hoboware graphical program. Unfortunately, Hoboware stores the extracted data in their proprietary format, locked away from any other program. I eventually discovered the configuration setting that automatically saves the data in CSV format, but I didn’t find that until rather late in the game, didn’t always set it with new versions, and it seems their CSV format has changed slightly over the years. Thus, one of my to-do items is to manually process the remaining Hoboware files to produce the corresponding CSV files, then convert those into a standard format that’s useful with, say, Gnuplot.

The intent is that I can simply concatenate all the CSV data files for a given sensor, run them through a Bash script to sanitize the data, plot what emerges, and then maybe slice-and-dice the data a few different ways. The less manual processing this requires, the more it will get done…

But the first step is to show that something emerges from the data, so here’s the last year of data (recorded in 2012, which includes a bit of 2011 and not quite up to the end of 2012) from the logger that’s been monitoring the air temperature of the Basement Laboratory and the temperature at the house water inlet. I assume the minimum water temperature on the pipe at the basement wall tracks the ground temperature four or five feet down from the surface; more on the hardware behind the data in a while.

The fuzz on the purple trace shows the relatively rapid temperature variation as we draw water from the supply: it falls as water moves into the house and rises as still water warms. The inlet always remains cooler than the air temperature, because it’s cemented to the wall, but a closer look (again, in a while) shows a nice exponential curve. The thin straight-line sections show gaps in the data record: sometimes I forget to do my monthly science for a few days or weeks.

An extract from the CSV files, including some data not plotted above:

"Plot Title: Town Water Inlet "
"#","Time, GMT-04:00","Temp, °F","RH, %","Temp, °F","Host Connected","Stopped","End Of File"
1,09/25/2012 09:20:00,66.344,58.707,64.632,,,
2,09/25/2012 09:25:00,66.173,57.579,64.459,,,
-- snippage --
12962,11/09/2012 09:25:00,60.174,54.301,56.685,,,
12963,11/09/2012 09:28:48,,,,Logged,,
12964,11/09/2012 09:28:55,,,,,Logged,Logged
-- snippage --
"Plot Title: Town Water Inlet "
"#","Time, GMT-05:00","Temp, °F()","RH, %()","Temp, °F()","End Of File()"
1,11/09/12 08:35:00 ,64.247,52.282,56.728,
2,11/09/12 08:40:00 ,63.304,51.465,56.728,
-- snippage --
14473,12/29/12 14:35:00 ,56.599,51.454,48.895,
14474,12/29/12 14:40:00 ,56.599,51.485,49.116,Logged

A touch of sed can handle the reformatting I’ve seen so far:

• Convert headers to comments: sed 's/^\"/#&/'
• Convert non-data events to comments: sed 's/^.*Logged/#&/'
• Remove spurious trailing blanks in data fields: sed 's/ ,/,/'

Here’s the Bash and Gnuplot source code that produced the graph, complete with cruft that may come in handy later:

#!/bin/sh
#-- overhead
export GDFONTPATH="/usr/share/fonts/truetype/"
base="${1%.*}"
echo Base name: ${base}
tfile1=$(tempfile)
tfile2=$(tempfile)
ofile=${base}.png
echo Input file: $1
echo Temporary files: ${tfile1} ${tfile2}
echo Output file: ${ofile}
#-- prepare csv Hobo logger file
sed 's/^\"/#&/' $1 > ${tfile1}
sed 's/^.*Logged/#&/' ${tfile1} > ${tfile2}
#-- do it
gnuplot << EOF
#set term x11
set term png font "arialbd.ttf" 18 size 950,600
set output "${ofile}"
set title "${base}"
set key noautotitles
unset mouse
set bmargin 4
set grid xtics ytics
set timefmt "%m/%d/%Y %H:%M:%S"
set xdata time
set xlabel "Week of Year"
set format x "%W"
#set xrange [1.8:2.2]
#set xtics 0,5
#set mxtics 2
#set logscale y
#set ytics nomirror autofreq
set ylabel "Temperature - F"
#set format y "%4.0f"
set yrange [30:90]
#set mytics 2
#set y2label "right side variable"
#set y2tics nomirror autofreq 2
#set format y2 "%3.0f"
#set y2range [0:200]
#set y2tics 32
#set rmargin 9
set datafile separator ","
#set label 1 "label text" at 2.100,110 right font "arialbd,18"
#set arrow from 2.100,110 to 2.105,103 lt 1 lw 2 lc 0
plot	\
    "${tfile2}" using 2:3 with lines lt 3 title "Air", \
    "${tfile2}" using 2:5 with lines lt 4 title "Water"
EOF