The WS2812 under the 5U4GB full-wave rectifier tube went into pinball panic mode:

It’s been running more-or-less continuously since late 2016, so call it

Because I’d be crazy to replace it with another likely-to-fail WS2812, I had to remove both of them before installing SK6812 RGBW LEDs and updating the Arduino Nano.

Unfortunately, I did a really good job of bonding the side light to the tube with epoxy:

The last tube manufacturing step involved flashing the getter onto the tube envelope, so as to remove the last vestige of air. Admitting air oxidizes the getter:

It was such a pretty tube, too …

My “exhibit” at the MHV LUG Mad Science Fair consisted of glowy and blinky LED goodness, with an array of vacuum tubes, bulbs, and the WS2812 and SK6812 test fixtures:

MHVLUG Science Fair – Chastain – highres_463020980

They look much better without a flash, honest. The cut-up cardboard box threw much needed shade; the auditorium has big incandescent can lights directly overhead.

Anyhow, what with one thing and another, the two LED test fixtures spent another few dark and cool days in the Basement Laboratory. When I finally plugged them in, the SK6812 RGBW LED array light up just fine, but three more WS2812 RGB LEDs went toes-up:

WS2812 LED test fixture – more failures

That brought the total to about 8 (one looks like it’s working)  out of 28: call it a 28% failure rate. While WS2812 LEDs don’t offer much in the way of reliability, running them continuously seems to minimize the carnage.

So I wired around the new deaders and took that picture.

Flushed with success and anxious to get this over with, I sealed the tester in a plastic bag and tossed it in the freezer for a few hours …

Which promptly killed most of the remaining WS2812 chips, to the extent even a protracted session on the Squidwrench Operating Table couldn’t fix it. When I though I had all the deaders bypassed, an LED early in the string would wig out and flip the panel back to pinball panic mode.

It’s not a 100% failure rate, but close enough: they’re dead to me.

As the remaining WS2812 LEDs on the various vacuum tubes and bulbs go bad, I’m replacing them with SK6812 RGBW LEDs.

For whatever it’s worth, freezing the SK6812 tester had no effect: all 25 LEDs lit up perfectly and run fine. Maybe some of those chips will die in a few days, but, to date, they’ve been utterly reliable.

The two knockoff Neopixel test fixtures went dark while their USB charger accompanied me on a trip, so they spent a few days at ambient basement conditions. When I plugged them back into the charger, pretty much the entire array lit up in pinball panic mode:

WS2812 LED – test fixture multiple failures

Turns out three more WS2812 chips failed in quick succession. I’ve hotwired around the deaders (output disconnected, next chip input in parallel) and, as with the other zombies, they sometimes work and sometimes flicker. That’s five failures in 28 LEDs over four months, a bit under 3000 operating hours.

For lack of a better explanation: the cool chips pulled relatively moist air through their failed silicone encapsulation, quietly rotted out in the dark, then failed when reheated. After they spend enough time flailing around, the more-or-less normal operating temperatures drives out the moisture and they (sometimes) resume working.

Remember, all of them passed the Josh Sharpie Test, so you can’t identify weak ones ahead of time.

A second WS2812 RGB LED in the test fixture failed:

WS2812 LED – test fixture failure 2

The red pixel in the second row from the top sends pinball panic to the six downstream LEDs (left and upward). Of course, it’s not consistently bad and sometimes behaves perfectly. The dark row below it contains perfectly good LEDs: they’re in a dark-blue part of the cycle.

The first WS2812 failed after about a week. This one lasted 7 weeks = 50-ish days.

The encapsulation seal went bad on this one and, for whatever it’s worth, the remainder still pass the Sharpie test. Perhaps the LEDs fail only after heat (or time-at-temperature) breaks the seal. Assuming, equally of course, the seal left the factory in good order, which seems a completely unwarranted assumption.

The failed WS2812 pixel remains defunct:

WS2812 array – failure 1

Attach scope probes to its data input and output pins (with the fixture face-down on the bench):

WS2812 LED – fixture probing

The output no longer comes from the Land of Digital Signals:

WS2812 Array Fail 1 – in vs out

I immediately thought the broken bits occupied the first 24 bit times, when the WS2812 controller should be absorbing those bits from the incoming stream. The vertical cursors show the failed bits occupy 54 µs = 40-ish bit times at 800 kHz (or you can count them), so it’s worse than a simple logic failure.

A closer look:

WS2812 Array Fail 1 – in vs out – detail

At least for those bits, neither output transistor works well at all. On the other paw, the output shouldn’t even be enabled for the first 24 bits, so there’s that to consider.

Lo and behold, it also fails the Josh Sharpie Test:

WS2812 LED – test array failure 1 – ink test

You may recall it passed the leak test shortly before I assembled the test array a month ago. Evidently, just few days of operation suffices to wreck the seal, let air / moisture into the package, and kill the controller. Not a problem you’d find during a production-line test (assuming there is such a thing), but it should certain appear during the initial design & production qualification test phase (another assumption).

Weirdly, a day after taking that photo, the controller began working perfectly again and the LEDs look just like they should: there is no explaining that!

A few days after epoxying a replacement WS2812 RGB LED into the base of the 21HB5A and, en passant, soldering a 3.5 mm plug-and-jack into the plate lead for EZ removal, the top LED failed.

21HB5A – Audio plug cable

In this case, it also failed the Josh Sharpie test with bad encapsulation sealing:

WS2812 LED failure – ink test patterns

Here’s a view from another angle, with a warm-white desk lamp for a bit of color:

WS2812 LED failure – ink test patterns – 2

Those patterns took a few days to appear and also showed up in some, but not all, of the previous failing LEDs.

Although I have no idea what’s going on, it’s certainly distinctive!

An envelope of RGBW LEDs, allegedly with SK2812 controllers, has arrived from a different eBay supplier, so it’s time for an upgrade.

Well, that didn’t take long:

WS2812 array – failure 1

The red spot in the next-to-bottom row of the test fixture (*) marks a failed WS2812 LED. All of the LEDs above it, plus the LED just to its left, are in pinball panic mode: random colors flicker across the panel as the LED’s controller transmits garbled data and the downstream LEDs pass it on.

This failure provides several bits of information:

• The LED sees the same power supply as all the rest, so it’s not a power thing
• The LED gets data from the adjacent WS2812, so it’s not an Arduino output thing
• It failed after about four days = 100 hours of continuous operation

I connected the previous LED’s output (#6) to the next one’s input (#8), so the failed LED (#7, now with output disconnected) continues to flicker, but doesn’t influence any of the downstream LEDs.

(*) The LEDs are daisy-chained from lower right to upper left, row by row, so that’s LED #7 of 28.