The Smell of Molten Projects in the Morning

Ed Nisley's Blog: Shop notes, electronics, firmware, machinery, 3D printing, laser cuttery, and curiosities. Contents: 100% human thinking, 0% AI slop.

Category: Photography & Images

Taking & making images.

  • Recovering JPG Images From Damaged Flash Memory

    My DSC-F717 just crashed as I took a picture in the Basement Laboratory; it stalled while writing a picture to the memory stick. This camera is known to have problems with its ribbon cables and I’ve fixed it a few times before, but right now I have other things to do. Who knows? Maybe it’s a different problem altogether.

    Thus, the pertinent question is how to grab the image files from the Memory Stick, even with a damaged filesystem. This trick will work with any memory device, so it’s not just a Memory Stick thing.

    The camera crashed with the Memory Stick activity LED stuck on. Turning the camera off didn’t work: it jammed in the power-down sequence. So I poked the Reset button, which snapped the camera back to 1 January 2002, just after midnight. Alas, it now couldn’t read the Memory Stick, which meant either the filesystem is pooched or the camera’s card socket has gone bad again.

    Mmm, do you know where your camera’s Reset button is? It might not even have one, in which case you just yank the battery. If you can yank the battery, that is.

    Anyhow…

    With the camera turned off: extract the memory card, poke it into a suitable USB card reader, and poke that into your PC (running Linux, of course).

    If the filesystem isn’t too badly damaged, you’ll probably get a popup asking what to do with the new drive (the memory card). Dismiss that, as you don’t want anything writing to the memory without your explicit permission, which you won’t give.

    Figure out which device corresponds to the card and which partition to use:

    dmesg | tail
    [29846.600524] sd 5:0:0:2: [sdd] 1947648 512-byte hardware sectors (997 MB)
    [29846.606022] sd 5:0:0:2: [sdd] Write Protect is off
    [29846.606030] sd 5:0:0:2: [sdd] Mode Sense: 23 00 00 00
    [29846.606033] sd 5:0:0:2: [sdd] Assuming drive cache: write through
    [29846.608748]  sdd: sdd1
    

    In this case, the Memory Stick was intact enough to have a valid partition table, so it could be successfully mounted. However, you don’t want any attempt to write the data, just to keep a bad situation from getting worse. You do that by mounting the device read-only:

    sudo mount -o ro /dev/sdd1 /mnt/part

    You’ll need a suitable mount point; I created /mnt/part early on to hold manually mounted disk partitions.

    The FAT filesystem seemed to point to valid files, but attempting to display them didn’t work. So unmount the device before proceding:

    sudo umount /dev/sdd1

    Switch to /tmp or anywhere you have enough elbow room for a few files the size of the memory card.

    Run dd to make a bit-for-bit copy of the entire drive:

    sudo dd if=/dev/sdd of=memstick.bin bs=1M

    The bs=1M should dramatically speed up the transfer; the default is, IIRC, 512 bytes.

    After this, everything you do uses memstick.bin, not the memory card itself. If you’re paranoid like me, you’ll make a copy of the file before you do anything hazardous. You could even make two copies directly from the memory card and compare them, which might be instructive.

    To find your precious JPG images among the rubble, look for their magic Exif headers:

    strings -t x memstick.bin | grep Exif
      68006 Exif
     258006 Exif
     680006 Exif
     848006 Exif
     a18006 Exif
     bf0006 Exif
     e00006 Exif
     fe8006 Exif
    11e8006 Exif
    1420006 Exif
    ... snippage ...
    

    Because the strings search ignores the (presumably broken) FAT directory structure, you’ll find all the image files on the drive, whether or not they’ve been deleted, partially overwritten, or whatever. This is great for forensics and terrible if you’ve got something to hide. You have been warned.

    The -t x parameter returns the string’s starting offset in hex, which you need to find the actual starting offset of the file: it’s 6 bytes lower! Your mileage may vary, so be prepared to fiddle around a bit.

    You could write a bunch of code to parse the JPG header and extract exactly the number of bytes required, but this is no time for subtle gestures. I just yank out whatever the largest possible image file could be, because image-processing programs only process the valid stuff anyway. So, knowing that this camera produces images in the 2.4 MB range, this extracts the first image:

    dd if=memstick.bin of=image.jpg bs=$((16#1000)) count=1K skip=$((16#68))
    

    The $(( … )) notation evaluates the numeric expression within and the 16#… notation expresses a hexadecimal value.

    Soooo, bs=$((16#1000)) says that the blocksize is 4096 bytes, which you can deduce from the fact that all the Exif headers start 6 bytes from a multiple of 0x1000. Again, your camera may do things differently, but the general notion should get you started.

    If you’re fussy, you’ll note that the headers are actually on multiples of 0x8000 bytes, but using 0x1000 means you can read the high-order digits right off the strings dump. Why make things more complicated than necessary?

    Given a 4K blocksize, count=1K extracts a 4MB chunk: 1024 blocks of 4096 bytes each. That’s larger than the largest possible image file from this camera; adjust it to suit whatever you expect to find. Don’t be stingy, OK?

    The skip=$((16#68)) says to begin extracting data 0x68 blocks into memstick.bin. You read that value directly from the strings output, which is easy enough.

    You could write a tidy Bash script to eat the strings values and spit out the corresponding file chunks. I had few enough images this time to just do it manually, which beats having to debug some code…

    Good luck!

  • Quick-and-easy IR-passing / Visible-blocking Optical Filter

    Gel filters - normal visible light
    Gel filters – normal visible light

    When you don’t need high optical quality for an IR filter, you can superimpose red and blue stage-lighting filters: pure black to the eye, transparent to IR.

    You know they’re IR-transparent because they’re generally snuggled right up against huge incandescent bulbs: if the filter material absorbed any IR, it’d burn right up.

    I’ve used Lee Filters Congo Blue (181) and Primary Red (106) to good effect. They may be available from a stagecraft outlet near you, but around here that stuff is a mailorder deal. You’ll get a lifetime supply, so maybe you can pass some out to your cronies; techies always enjoy odd presents like that.

    A more optically flat (and durable and expensive) option would be a photographic-image-quality glass IR filter suitable for camera mounting. I got one after I dropped a homebrew plastic filter down a sewer grate.

    For examples, go to Adorama, click on Filters in the left column, then select Infra-Red Filters, then maybe refine the search to the cheaper Hoya brand before your budget runs away in fear.

    Gel filters - IR view with visible light
    Gel filters – IR view with visible light

    Make sure your camera doesn’t have an IR blocking filter behind the lens. I think most consumer-grade digital cameras do have an IR-blocking filter and most video cameras don’t, but I’m sure those general rules don’t hold in all cases. Indeed, I bought a Sony DSC-F717 specifically for its IR mode; fortunately, the CCD sensor failed shortly before the factory recall ended.

    The pictures show the same scene under normal lighting, with the camera set to its IR mode, and IR mode with an IR filter in front of the lens.

    The gel filters appear dark-gray in the middle image because the camera sets the exposure (1/60 f2.4 ISO100) based on the visible light entering the lens. They’re transparent in the bottom image because the exposure (1/30 f2.4 ISO1000) is based on only the IR illumination, which is pretty dim. The gratuitous greenish cast is how Sony reminds you that the image was in IR mode

    Gel filters - Pure IR view
    Gel filters – Pure IR view
  • Extracting Digital Camera Multi-burst Images

    Multi-burst image of trebuchet firing
    Multi-burst image of trebuchet firing

    Digital camera, at least the ones I have, include a “multi-burst” (Sony DSC-H5 and DSC-F717) or “multi-continuous” (Casio EX-Z850) shutter mode that takes a bunch of pictures in quick succession, then combines them into a single JPG image file.

    The Sony cameras create a 4×4 array. This image of a small trebuchet comes from the F717 and is 1280×960, so each sub-image is 320×240. The time between images is 1/30 second and the shutter speed is 1/125 second.

    Extracting the sub-images is trivially easy with the ImageMagick convert function:

    convert -crop 320x240 dsc02594.jpg shot-%02d.jpg
    Sub-image shot-11.jpg from the sequence
    Sub-image shot-11.jpg from the sequence

    You must specify the size of the sub-images to extract, which you can determine by RTFM or simple division, and convert extracts all the tiles into files named, in this case, shot-00.jpg through shot-15.jpg. The files appear in left-to-right, top-to-bottom order, which is most likely the sensible way for cameras to store them.

    The C printf-style format string %02d forces two-digit sequence numbers. You can omit that and the sequence will start with shot-0.jpg, but you must then contend with the usual hassles of shot-1.jpg and shot-10.jpg.

    Can you tell the designers were computer geeks?

    With 16 separate images in hand, you can have your way with them, using all your usual image-manipulation tools.

    ImageMagick can convert the images into an animated GIF:

    convert -delay 50 shot-*jpg shot-ani.gif
    Animated GIF from separate images
    Animated GIF from separate images

    That’s nigh onto 7 MB of image, which seems excessive for what it is, but there you have it. Obviously, you can de-res the images to fit the space available.

    The -delay 50 option should set the frame delay to 50 ticks at the default 100 ticks per second, but some display software ignores the frame rate within the file. Assuming, that is, that the usual spam filters don’t swat animated GIFs right out of the bitstream.

    You can also convert the images into a movie, as I discussed in more detail there. The ffmpeg program does a fine job of it:

    ffmpeg -r 3 -i shot-%02d.jpg shot.mp4

    Actually, convert can do that all by itself if you install mpeg2encode.

    One cannot upload movies to one’s free WordPress blog without buying a space expansion, so you’ll have to take my word that it works.

    The Sony cameras provide control over the interval between the images, allowing 1/30, 1/15, and 1/7.5 second intervals, but the Casio evidently just does the best it can. If you know the interval, you can determine interesting things like velocity and acceleration, so that’s something to look for when you’re buying a camera.

    Perhaps you can calibrate your camera using a pendulum?

    Memo to Self: Use the tripod!

  • Photo Tweakage: Mouse Tunnels in the Snow

    Mouse tunnels in the snow
    Mouse tunnels in the snow

    As the snow cover melts away in the spring, you discover how much activity has been going on.

    The mice make elaborate tunnels with spaces for seed stores, latrines, and numereous secret entrances near the bushes. For a few months, at least, they can scamper all over the yard without worrying about becoming snacks for owls and hawks.

    This kind of picture requires a bit of tweakage, because the default camera settings deliver an essentially gray picture with no contrast. The usual auto-exposure settings assume a more-or-less neutral background, so the camera adjusts the exposure to deliver a neutral result. Unfortunately, you really want most of the background to be white, so the default snow image will be grossly underexposed.

    Set the camera to overexpose the picture by 1 or 2 EV; if your camera has a histogram display, adjust the exposure to put that huge bump on the white end close to the right side of the histogram. It helps if you frame the picture before doing this, as the LCD monitor will be pretty much retina-burn white.

    Take the picture and get it into your PC.

    GIMP Level Adjustment Window
    GIMP Level Adjustment Window

    Now, in your favorite photo-editing software (The GIMP on our desktops), adjust the photo’s levels & contrast. This screen shot shows the logarithmic histogram; the linear one is basically just one peak 3/4 of the way to the right side with a little grass on the rest of the chart. The two little buttons in the upper-right choose linear or log.

    Drag the white point (the teeny white triangle on the right, just below the histogram) until it’s just a bit to the right of the abrupt dropoff’s foot. That sets the whitest part of the picture to real white, not half-a-stop-down light gray.

    You can do the same with the black point (the black triangle on the left), setting it to just left of the black dropoff. In this case, we have some genuinely dark areas, so leave it alone.

    Now, the key part: drag the middle triangle to adjust the gamma. Move it rightward to darken the overall image by decreasing the gamma and emphasize small differences on the bright end of the histogram. That makes the tunnels pop out of the image, although it also tends to make the snow look very contrasty.

    Don’t go overboard with this sort of thing, but a little adjustment can reveal details and bring pictures back to life. It’s not for Ansel Adams quality pix, that’s for sure.

    Here’s possibly more than you want to know about levels & gamma & contrast, but with much better illustrations and more descriptions: Unai Garro’s Blog. He has other useful tutorials, too.

  • Digital Camera Resolution: It’s Not The Dots

    Overview of the scene - 8 MP image
    Overview of the scene – 8 MP image

    A rule of thumb for cameras is that more glass is better, so I’ve always been skeptical of those little bitty chips o’ glass in front of all those pocket-sized camera CCDs.

    A while ago I took some shots with my Casio EX-Z850 at various resolutions, chopped out a chunk with some good detail from the middle of the images, and resized (cubic interpolation) the smaller ones to match the 8 MP image. The exposure was automagic: 1/200 sec at f/7.4. Zoomed as far as the optical zoom would go, hand-held ’cause at that shutter speed it’s OK, sharpness and contrast one click higher than the default.

    As nearly as I can tell, there’s only a slight difference between 8 & 6 MP, enough to be noticeable at 4 MP, and it’s getting on toward icky at 2 MP. The picture isn’t any bigger because I wanted to preserve the actual dots in the 8 MP image.

    Resolution Test - Shield
    Resolution Test – Shield

    Notice the weird color gradations in the shields with bright primary colors. This was hand-held, so the camera reached the same image conversion conclusions each time, starting with different physical pixels. A bit of rummaging turns up the Island Coffee Co logo at http://www.islandcoffee.net/.

    Just to show that resolution doesn’t matter in real life, check out the snail. Zoomed all the way out, macro focus, 1/125 f/2.8 (wide open), hand held, resized as above.

    Snail
    Resolution Test Images: Snail

    As nearly as I can tell, the 2 MP image is the only one that’s noticeably worse and it’s still just fine. The other three are different, but even the 2 MP one isn’t bad.

    Moral of the story: lots o’ dots is good, but beyond a few MP it just doesn’t matter. I think 4 MP is the sweet spot, based on an entirely insufficient sample, with 8 MP if you’re planning tight cropping. Otherwise, save 2 MB per image: 8 MP -> 4-5 MB, 4 MP -> 2-3 MB.

    The other High Truth: lots of light is a Good Thing. You get really crappy pictures when there’s not enough light, but it seems that’s where all the interesting stuff happens.

    Update: Uploaded the PNG version of the shield resolution test, as the JPG was fairly crappy, plus 8 MP overviews of the scenes. The text is antialiased and looks awful when you zoom in, but the whole point is to show the actual dots produced by the camera: there aren’t that many dots for the letters.

    Snail overview - 8 MP image
    Snail overview – 8 MP image
  • Pocket Camera: Griptivity Thereof

    Casio EX-Z850 and homebrew case
    Casio EX-Z850 and homebrew case

    I carry a small camera with me at all times and find it invaluable for recording details and documenting events; now I never say “I wish I had a camera!”

    This one is a Casio EX-Z850, which trades off nearly everything in favor of compact size. It has great battery life, enough resolution (the optics could be better), and manual controls (so it serves nicely as a microscope camera). It’s obsolete, of course, but you get the idea.

    I have bigger & better cameras, but this one is always with me and that counts for nearly everything. The camera in a cellphone or PDA is not the same as a real camera.

    Unfortunately, the thing has the griptivity of a bar of soap: all stylin’ metal and plastic. The black nubbly surfaces in the photo are my idea of a Good Thing: chunks of stair-tread tape providing enough traction that the camera no longer flies out of my hand with the greatest of ease.

    Despite that, I always slip the lanyard over my wrist when I take it out of my pocket; often I do that before removing it from the case. That nervous tic saves me the cost of a new camera about twice a year.

    If your camera fits into a desktop charging / USB cradle, as this one does, make sure you don’t stick the tape where the cradle fits against the camera. It’s really tough to peel off after the adhesive sets up…

    Mary made that nice packcloth (she says “Cordura“) case, with a fuzzy fleece liner facing the LCD panel. The hook-and-loop closure is a tad noisy in quiet places, but it’s better than buttons or a zipper for this application.

    I’ve learned to not keep tissues in the same pocket as the camera.

  • Daily Yard Picture

    5 November 2008
    5 November 2008

    Being that sort of bear, I took a picture of the back yard from our patio every day at 7 am wall-clock time. DST/EST changeovers threw their usual monkey wrenches into the mix, not to mention my lack of attention to the camera’s internal clock settings, but I eventually got 321 pictures of the same scene at more or less the same time of day.

    That’s all well and good, but this is the movie age…

    The plan: use ffmpeg or maybe mencoder to convert the still images into a movie.

    • Zero: copy the files to a unique subdirectory to protect the originals!
    • One: sort & rename by date
    • Two: resize images
    • Three: convert to a movie
    • Four: . . . profit!
    10 November 2008
    10 November 2008

    I’d uploaded the files whenever I used the camera for something else, so the actual file dates were fairly well scrambled and didn’t correspond to the EXIF data inside the image file. Digikam‘s batch file rename operation can sort out the files in ascending order of EXIF date and rename them into something a bit more uniform & boring like 0001.jpg, which is vital for ffmpeg.

    I used the camera’s full resolution, which is much too large for video, so I created Yet Another Subdirectory called Smaller to hold the reduced-size images. Imagemagick‘s convert program then squishes them down:

    for f in *jpg ; do convert -verbose -resize 640x480 $f Smaller/$f; done

    You can smash them even further to get a teeny postage-stamp movie for your media player.

    Make the movie:

    ffmpeg -r 3 -i %04d.jpg daily-3.mp4

    The file specifier %04d must exactly match the filename sequence and a missing file will stop ffmpeg dead in its tracks. The file names coming out of your camera won’t work if they’re not exactly sequential, which is highly unlikely over the course of the year.

    You can use mencoder:

    mencoder "mf://*.jpg" -mf fps=10 -o daily800.avi -ovc lavc -lavcopts vcodec=msmpeg4v2:vbitrate=800

    Then it’s showtime! I’d upload it, but you don’t have a need to know for our backyard activiites.

    There, now, wasn’t that easy?

    I didn’t actually figure all this out from first principles, of course. The basics are out there if you rummage around for a while with the obvious keywords.

    Memo to self: affix a stable camera platform to the side of the house!