PhotoSweeper: Photo De-Duplicator for Aperture / iPhoto / Lightroom / etc.

January 20th, 2015 (trackback)

Since I started taking photos 15 years ago, I’ve amassed a pile of images. 25,000 or so photos sourced from various cameras, phones and a handful that were scanned and imported. They have been managed manually, in various flavors of iPhoto, the occasional random application, and — for the past few years — in Aperture. Copies of the photos have lived on various computers and, even, have been recovered from backups after a rare hard drive death (multi-point backup strategy FTW!).

End result?

25,000 images of which somewhere between 5% and 20% are duplicates. Many of them are straight up duplicates; copies of the same image with different filenames resulting from merging various libraries or importing source media to multiple destinations that were later merged. Some are more insidious. Somewhere, something decided to down-res a slew of images and re-import them. Somewhere else, something decided to re-encode all my JPEG images (before RAW) at the same resolution, but with much higher compression.

Slogging through all those images would take hours. Days, really, as it’d have to be done in my spare time. And, given that it is a task I’ve been avoiding — digging the whole deeper — for a decade+, clearly not going to happen soon.

Clearly, there has to be a better way. And there is!

A quick search yielded PhotoSweeper.

On first pass, it quickly eliminated all straight up duplicates where the actual contents were identical. This took less than ten minutes to do on those 25,000 images and it eliminated nearly 4,000 dupes (and triplets and the occasional quad).

The second pass is where this software really shines. I configured it to do a content comparison and flag any sets of images that were pretty close, but not necessarily exactly, similar. In this case, I used the “approximate, align and blur” method. That is, PhotoSweeper re-renders each image as a 144×144 grid of pixels, then blurs it slightly, and aligns the edges. The resulting icon-ized images are compared and any that are similar enough are flagged as potential dupes. It is then a matter of review-and-compare. The arrow keys are used to navigate and the return key to toggle whether or not the image will be trashed.

One click and all the identified dupes are dumped in the trash.

What would have taken days of tedium was reduced to less than an hour. Personally? I would have paid $50 — nay, $100 — for this and have considered it a bargain. It saved me that much time (frankly, it finished a task in short order I’ve been putting off for a decade) and now my remaining organization task is largely one of actually looking at, tagging, and categorizing the photos.

And sharing them with my family. Because that’s what it is all about (for me).



Alaska

January 6th, 2015 (trackback)
Grizzly bear (Ursus arctos horribilis)

Over the summer, we escaped for a while to Alaska via a trip put together by International Expeditions. This was our second trip with the company — our first being a week on the Amazon River (fun story, photo set on flickr) — and it certainly won’t be our last as International Expeditions does a fantastic job.

For the Alaska trip, we opted for the 11 day grand tour.

Beyond a fantastic tour of glaciers, a backcountry lake lodge, and Denali National Park, highlights included fantastic lodging in the most remote of places, incredible homemade meals, mind blowing scenery, and a wonderfully friendly group of random travel companions that quickly become a temporary family (with whom we wouldn’t mind crossing paths again!).

It was, no surprise, a great place to take photos and I’ve pushed an album to Flickr.

Alaska is both surprisingly diverse and, yet, equally as surprisingly limited in natural diversity.

On our tour, we ranged from a coastal rainforest to arctic tundra and even spent some time in the air flying around Mt. Denali.

And that covered a very tiny part of the state. It is also hard to fathom just how big Alaska truly is. Watching this glacier calve, we were nearly 1 mile away from it and yet it still seems to tower above us and there were still miles of glacier behind and above it! And beyond the top of that glacier? An ice field that is larger than most states in the lower 48.

Alaska is very different than anywhere else we’ve visited. Whereas the Amazon was full of a seeming infinite number of species of plant an animal, there are actually very few species in Alaska; only 150 or so native plants have been cataloged (with many variants within those particular individuals).

While in the rainforests of the equator, things rot with incredible speed, trees may lie on the ground in the Alaska rainforest for years because their simply isn’t the warmth or sunlit energy to support the decomposing agents. Thus, the Alaskan rainforest floor is a thick layer of, well, mulch on top of rock. Springy, almost magical to walk on. But no real dirt.

And some of the seemingly most innocuous and beautiful plants are quite deadly.

Even the mosquitos are different. In Alaska, the mosquitos are the top pollinator. All those incredibly yummy tundra blueberries, cranberries, and crowberries? Pollinated by mosquitos. Still annoying critters. But mosquito defense did lead to some interesting fashion statements.

A fantastic place to visit!


Icy Creek Pt 3: Ice Fractals

January 1st, 2015 (trackback)
Ice Patterns on Creek

Ice Patterns on Creek

Suspended Ice

While walking on the creek, Roger and I found lots of different patterns of ice on the creek. There simply wasn’t any smooth ice anywhere.

Fractals, fractals, everywhere.

Some were on the surface of the water. Some floated above. Some were clear, others were not. All were beautiful.

Ice Patterns on Creek



Icy Creek Pt. 2: A Physics Lesson (Newton’s Rings)

January 1st, 2015 (trackback)
Ice Rainbow
Ice Rainbow

As Roger and I were walking along the frozen creek we were (duh!) throwing rocks to break ice, stomping on the ice, and generally checking out all the neat patterns that naturally occur (including the ice lens and another post that will highlight the amazing fractals that occur naturally).

When looking closely at some of the cracks, I noticed rainbows. Rainbows everywhere! They were typically formed right along the cracks and I had assumed they were the result of stress in the ice.

Sort of.

They are actually, as @maclemon kindly shared on Twitter, examples of Newton’s Rings.

I’m not smart enough to grok the Wikipedia entry entirely, but armed with that, I was able to search for other discussions.

Here is someone who ran across them in their backyard and did a bit of research.

And here is a whole discussion of the phenomena with lots of studio photos.

The photos at the left are two photos of the same spot. The top photo is focused on the surface and the bottom is focused on the plane of the rainbow.

I may add some more photos if I happen to see some neat patterns. Because, clearly, more ice shall be broken.



Icy Creek Pt 1; Natural Lenses

January 1st, 2015 (trackback)
Frozen Ice Lens on Leaf

Roger and I went for a long walk near my parent’s house in Columbia, MO. Being the end of the year, it is freezing cold and still. Ideal weather to produce ice on the ponds and creeks of the area.

While walking on the creek, we noticed that a couple of leaves had formed little balls of ice on the end of them. The ice was perfectly clear and the shape was quite regular.

Almost lens like, really.

Roger Through an Ice Lens
Through an Ice Lens

Sure enough. It did act as a lens!

These were shot with a 100mm macro lens on a Canon 7d Mk II (my christmas present).



Walker’s Cicada: A Photo Study

July 16th, 2014 (trackback)
Walker's cicada (Tibicen pronotalis)

“Hey, Dad! There’s a cicada emerging over there.”, Roger said.

Watching the emergence of any critter that goes through a metamorphic stage is always fascinating. While Roger (and I) have raised many butterflies (and Roger has raised quite a number of other critters), we’ve never watched the emergence of one of the annual cicadas common to the Midwest.

When an insect emerges from its metamorphic container — be it a cocoon or, in this case, the harden shell of the larval stage — it is typically quite vulnerable.

If winged, there is a long period of pumping fluids into the wings and then letting them effectively cure in the open air.

It is a particularly vulnerable time exactly because the animal is in between forms and simply can’t move.

So! Photo study time!

Some of the original images at full resolution are really stunning. This is a beautiful creature.

In this first image, the critter has just emerged. The wings are just starting to unfurl.

Walker's cicada (Tibicen pronotalis)

At this point, the cicada has fully emerged, but the wings are only partially expanded.

I was very surprised at how stunningly neon green the wings are!


Walker's cicada (Tibicen pronotalis)

This is a closeup (rotated) of the expanding wing. Note that those wings will be largely transparent once they are fully expanded and cured. At this point, though, it almost looks like frost.

Walker's cicada (Tibicen pronotalis)

This just seemed like an obvious picture to take. :) From this angle you can’t see all the facial hair!

That picture will be at the end.


Walker's cicada (Tibicen pronotalis)

At this point, the wings are fully expanded, but not yet cured.

The neon is just stunning. Surprisingly so. Gorgeous.

Walker's cicada (Tibicen pronotalis)

One final shot. The cicada flew away shortly hereafter (or crawled).

I was stunned by the golden highlights on the “shoulder” and the hairs around the eyes!

What a beautiful creature!



Mossbrook Fire: 10 Days After Update

April 18th, 2014 (trackback)
Car Windshield After Fire

Lots of people are asking about various events related to the fire. This is a summary of the first 10 days of our adventure broken down as a series of short comments. A bit of a ramble in no particular order, I suspect, but here it is.

I chose the photo at left as a reminder of all the things we didn’t lose — no humans were hurt and all of the pets except one parakeet made it through unscathed — and that even such a destructive force as a fire can yield surprising beauty.


In the days that followed the event, the San Jose Fire Department sent crews around fairly often. Not just to ensure no hot spots remained, but also to use the event as a means of learning to fight such firees more effectively in the future. It was very interesting to be a part of the conversations as to how they’ll modify strategy in the future and what worked for this event. I’ll continue to collect photo streams and other information on the Photo Dump post (in fact, there will be a new stream on that post shortly after this is posted).

It was kind of funny how apologetic the fire fighters were about the destruction they caused in the house (which was quite minor compared to the destruction caused by the actual fire). I finally stopped the two that were walking me through after the fire, pointed out my kitchen window at the utter devastation of the house next door, and told them that saving my house was repayment 100 times over vs. any damage they may have caused.


Now that the laundry room — the inside portion of the house most severely impacted — has been cleared, it has become very clear just how close we were to losing most or all of the house. The roof is beyond charred. Take a log from your fireplace after a roaring fire? Yeah, that’s our laundry room ceiling.

While the wall burned through on the outside, it didn’t burn through the drywall. If we had the original thin wooden wall paneling in that room, it would be a very different picture. Drywall makes a good fire break.

Everything in the laundry room is a total loss save for any metal or ceramic pieces. The washer, dryer, and utility sink all partially melted.

The sliding door is gone, and the frame melted through. There is a pool of aluminum on the floor.

Water got into some of the slate floor tiles and caused them to explode as the water boiled. Not enough to need to replace the tiles, though.

We had California Closets based shelving/storage in the laundry room. It was OK, but sub-optimal. We are going to pay the difference between replacing that and fixing it properly and use this as an opportunity to fix it the way we want it. This also means we can fix the dryer vent and all the plumbing, both of which are… stupid.


Our foam roof likely also contributed to the preservation of the house. The straight tar/gravel original roof melts in a fire and drips, basically, raw fuel onto the fire below. While the covering on the foam is pretty toasty, and entirely gone in some spots, the closed cell foam underneath is fine. In fact, our roof should still be watertight. Apparently, that’ll be tested next week as there is rain in the forecast.

While the foam is intact, it won’t be for long. Because the cantilevered eaves are completely toasted, they have to be replaced. That means replacing at least 3x the length of the overhang on the other side to support the cantilever.

But they can’t simply be cut back to the first beam because that creates a hinge effect that weakens the structural integrity of the roof.

Thus, they roof decking will have to be cut back to the first, second, and possibly the peak on that slope of the house. Likely, it’ll be a mix of cut backs to try and preserve some of the wood.

Bottom line: The foam roof on that slope of our roof is coming entirely off.


Electrical Panel & Laundry Wall

The electrical infrastructure on the house is completely toasted. More likely than not, there will have to be channels cut through the foam on the roof all over the house to run new wiring pretty much everywhere needed.

Same goes for the water and gas, but that is much much simpler infrastructure than the spider web that is the electrical wiring throughout the house.

At least we’ll be able to fix our thermostat! And add an outlet here and there!


A temporary power pole has been installed in the back yard. But the City won’t grant a permit for hooking it up until some other bit of paperwork is completed. This is, apparently, a new requirement and our contractors are trying to figure out why.

We should have power on site in the next week. At the moment, we have extension cords running to two neighbor’s houses to power the various filters needed for the fish and to power the gigantic air filtration unit brought in by the cleaning company.

Yes, the neighbors will be able to bill us for power used and insurance will cover it. More importantly, by doing this we don’t have a generator in the neighborhood running 24/7.


The vultures and ambulance chasers have finally gone away. Within hours of the event and for days after, we had a stream of contractors and public insurance policy adjusters show up trying to convince us to hire them and sell us on the notion that our insurance company is The Enemy.

It was bad enough that I told two of them that stepping foot on my property would be considered trespassing as they were no longer welcome and they could take it up with the local police.

This isn’t to say that I wouldn’t be willing to use a policy adjuster, if it were really necessary. But, no, that isn’t happening unless there is some issue with the insurance company (so far, no signs at all that there will be). And I’m sure as hell not going to use one that showed up on my property after chasing down the news copters.



The insurance company (State Farm) has been, thus far, great to work with. Their general approach is to offer full solutions, never push any given provider, and allow us to hire whomever we want to do any particular bit of work.

So far, the recovery efforts involve the following providers:

    Jon R Crase Construction
    We were introduced to Crase on the evening of the fire. They are on the short list of companies that the fire department uses to secure a site once the flames are out. As well, Crase is used by State Farm to double-check whatever contractors one might hire. And, most importantly, they have Eichler experience. Representatives from Crase have been on site and have consistently gone well above and beyond any contractually expected services. And they have that Eichler experience.
    Servpro
    ServPro is doing a pack out cleaning & storage. That is, they are packing out everything in the house that was affected by smoke/fire, inventorying everything, cleaning anything that needs to be cleaned, and then storing it until the house is ready to be moved back into.

    For all intents and purposes, it is as if we are moving out and in to our own house.

    This includes everything in the garage. All those nuts, bolts, screws, nails, and hardware that Roger and I have been collecting over the years? Yeah. Packed. Inventoried. Cleaned. And eventually returned.

    Custom Craft Urethane
    Keith Nokes of Custom Craft, who did our roof after the last remodel, was on site with ladder up before we had even started to consider how to pursue reconstruction. He wanted to see what’s what and offer any information he could. He immediately volunteered that he would want to be present for any roof work regardless of who we hired (some insurance companies would insist on particular people to do certain tasks). Yeah… no… Keith / Custom Craft will be doing our roof. Period. End of story. There is no one else we would remotely consider.
    Horizon Energy Systems
    While the solar wasn’t damaged, it is going to have to come off the roof for the reconstruction as the roof under it will be mostly replaced. Horizon installed it in the first place and did great work (the linked post has some insight into the madness of an Eichler roof). Yes, there were some significant challenges to the original installation, but Horizon has since modified their installation procedures because of the mis-adventures on our roof. Out of pocket, we’ll be adding additional rails for mounting more panels (but will hold on adding panels until we have electricity again).

What’s next?

Unlike a remodel, half of the demolition was unplanned and performed by a the monster that is an uncontrolled fire.

Thus, none of the planning that would normally have taken place prior to applying the first SawzAll has occurred.

The immediate next steps — now that the clean up is largely done to the point that doesn’t require demolition — is to secure the various permits with the City of San Jose. To that end, the City sent out an inspector to assess damage, write a report and put it on file. With that in place, pulling the needed permits should be relatively straightforward. Should be.

We also need full design documents drawn up for the floor plan of the house. On these will go the schematics for any work to be done. Because the spiderweb that is the electrical is destroyed at the panel, it is likely that the work will reach all corners of the house.

Given that the roof will be torn up, the walls redone, the laundry room rebuilt, etc… we’ll likely also use this as an opportunity to fix a few things here and there.



Mossbrook Fire: How can you help?

April 13th, 2014 (trackback)

They are questions that hav come up over and over; “How can I help?”, “What do you need?”, “What can we do?”.

The generosity and support of our neighbors, our friends, and our community have been truly amazing. We are surrounded by kind, generous, and awesome people through which we would easily have many places we could stay, temporary homes for our dogs, and all the support in the world.

But, really, we — the Bumgarner family, specifically — are fine. We, through an odd coincidence, have a house to live in for the near term and our insurance company’s response has been incredible (enough so, that a whole post will be devoted to the insurance process).

The two neighbors that lost their houses have places to stay, too. While there losses are total, their immediate needs are covered (seriously — the Bumgarner family got off easy in this one!).

And, really, this is about the same story is it will be with any disaster where the families involved are both fully insured and live in a supportive community.

So, how can you help?

At the scene as it happens?

Get the person away from watching the destruction of all their worldly possessions. Take them into a home far enough away from the scene so as to not be able to directly watch, but close enough to still be available and “behind the lines”, so to speak.

A couple of neighbors did that for my wife, son, and dogs. It was a tremendous help. It provides a base of sanity and calm in between bouts of dealing with the chaos.

Just Be Available (But Don’t Interrogate)

Every person will process what has happened differently and at a different pace. Some won’t need much support at all, others may need lots. Some might need some very odd thing to make them whole again.

Providing the most basic of support through simply listening will be of great help.

Avoid interrogation. No, in the 48 hours after the event, we don’t know how long it’ll take to rebuild our house, we don’t know what the neighbors’ plans are and, no, it really isn’t helpful to suggest what the neighbors might do one way or another. We — all of us, neighbors whose houses were unaffected, too — are simply trying to get to tomorrow with one less set of variables hanging over our heads.

Host or Provide a Home Cooked Meal

On the night after the fire, a friend of ours showed up with a steak, potato and asparagus dinner. She set the table, served the meal and cleaned up. It was, truly, a wonderfully relaxing and therapeutic event.

Last night, another friend of ours brought over a gigantic pot of jambalaya and we had dinner with the family that lost their home. Their comment? “This is the first home cooked meal we’ve had and it is wonderful.”

So, really, something as simple as inviting the impacted families for a simple meal or taking a meal to them will provide more comfort than you can imagine.

Donate to the Red Cross

The Red Cross was on scene the evening of the fire and had a triage center set up at the nearest school. They found lodging for the one family that needed it and triple-checked that the rest of us really did have a viable living plan. Within 24 hours, the Red Cross had agents in the area to see if anyone needed any kind of support, including counseling to deal with the trauma of the event. This included support for all of those living in the neighborhood.

Make Sure This Doesn’t Happen To You!

More likely than not, this was initially an electrical fire. The houses in our neighborhood were all built in the early 1960s and many have the original breaker box with the original wiring. In particular, all of the houses were built with Zinsco electrical panels. They aren’t safe.

So, yes, you should, right now, go out, have a look at your electrical panel, and replace it if it is more than a few decades old or otherwise unsafe.

Same goes for your gas infrastructure. Go have a sniff around your meter. Or ask that the gas company bring a sniffer and simply check. In the wake of the fire, PG&E went door to door with a sniffer and checked for leaks. Sure enough, at least one neighbor had a leaky connection near the meter.


Mossbrook Fire: Photo & News Dump

April 12th, 2014 (trackback)
IMG 2151

Being in the middle of a multi-home fire event with extensive destruction, there is a large pool of people that want to know what happened, how it happened, and progress during cleanup. Beyond the obvious family members, this also includes the various insurance company agents, fire investigators, and the fire department (who has regularly sent out crews to the site to both ensure no flare ups, but also to talk through the fight with those not there to ensure their techniques are refined in the future).

This post is for all of them. And it is to highlight the absolutely fantastic neighborhood that we live in.

Our neighbor, Al Boyden, was one of the first on the scene with a camera (that is his picture to the left). His iCloud photo stream includes photos from very near the beginning of the fire through to the photo to the left, when both the neighbor’s house was fully in flames and the fire trucks were just rolling up. I’ll have to ask Al to update the photos with the original timestamps. It is astonishing how fast an Eichler goes from “singed in the corner” to “full blown, total destruction, flames”.

My photo stream contains a series of photos taken after the fact that include documentation of both the destruction wrought and the cleanup process. Eventually, it’ll also include all the photos of the reconstruction process (which will be quite involved and, in some respects, even more involved than our remodel because part of the roof is going to be replaced).

Craig Allyn Rose appears to be an official photographer for the SJFD and has an incredible gallery of images from the event. The photos are primarily taken nearer to the two neighbors whose houses were destroyed. I’ve reached out to Craig to see if there is any additional information he would like added to this post. (Thank you, Craig, for your efforts and for sharing.)

Another professional photography on the scene was Chris Smead of Chris Smead Photography, who also works with the Fire Department.



Mossbrook Fire: The Media Machine Kinda Sucks

April 11th, 2014 (trackback)
Mossbrook Fire Panorama

90 more seconds.

A gentle wind blowing from the opposite direction.

That is all that made the difference between our house being the and one house was severely damaged vs. one of the houses that were destroyed in all the news reports.

Well, it would be if any of the news reports were correct. Even individual reports started out claiming that 3 houses were destroyed with a paragraph or two later claiming that a bulldozer arrived to tear down the two houses that were destroyed with one house receiving cosmetic damage (Uh… that my house’s entire electrical subsystem is melted and the laundry room is a black pit of doom doesn’t seem cosmetic).

Here are some facts. This is not a complete set, but merely a set of facts that are all quite directly addressed at the wild inaccuracies in the press:

  1. 1 house was destroyed completely, 1 house was gutted save for the garage, and 1 house — our house — was heavily damaged on the side facing the house that was destroyed.
  2. Of the two destroyed homes, only one is down and that is because it was collapsing once the fire was under control. The bulldozer was used to pull it down further so that hot spots could be uncovered so they could be doused with foam (in fact, one such hot spot reared up and produced 6′ flames at midnight, nearly 7 hours after the blaze was brought under control).
  3. Of the two destroyed homes, the families have lost everything.
  4. Burst Propane Tankes
  5. There were no gunshots. There were explosions. At least three of them in fact. And the propane tanks that were the source of [most of?] those explosions are currently in my backyard (but the firefighters moved them into my yard to get ‘em out of the way).
  6. The fire was not caused by a gas leak, nor was there a gas leak during the blaze. The fire was fueled by gas because the fire destroyed both gas pipes in the house and then the meter at the front of the house. So, yeah, that is sort of a gas leak. But not really.
  7. It took a while to turn off the gas because you can’t just turn off a whole neighborhood without the gas in the pipes feeding the flames for quite a while. Turning off the gas to a single home in an older neighborhood involves either turning off the gas at the meter (which was in flames) or digging up the connection to the gas mains and disconnecting the pipe there.
  8. I found Tyson, the dog, after the fire when we were first allowed to go into our house. Or he found me. When I walked into the backyard, he was standing in the backyard of his (now destroyed) home. I’m not entirely sure where he bunked down during the fire (most likely in the shadow of a stump), but there he was without even a bit of singing. He greeted me with a wag and bolted out the front door to find Paul, the kid who rescued him in the first place and for whom he shows a loyalty that only a dog can exhibit. Fantastic dog, by the way.
  9. There was no large scale evacuation. The immediate neighbor next to the fire was evacuated from her home. Beyond that, neighbors voluntarily left and stayed behind the lines of defense established by the fire department. Most of us hung out in a neighbor’s yard, generally being very supportive (our neighborhood is awesome), and sharing a bit of wine.

Beyond the simple fact that having your house damaged or destroyed by fire really really sucks, it is quite odd to be one of the unwilling subjects of a major news story. We heard tale from neighbors that their families in other states were calling them because they saw a blurb on the news.

The odd part is how every single article out there got one or many of the bullet points above wrong. I chose that particular set of facts because I read a story that incorrectly made a claim in conflict with said facts. Things move fast and I understand how such errors might happen, but to see a single article claim both that all three houses were destroyed and one house was destroyed and two houses had minor damage is indicative of how little importance is given to accuracy.