Locus Lingua - Jeremy Irish's Blog

Oven Roasted Jeremy with Sage

2009-07-09T12:11:00.000-07:00

This Sunday I depart for the hot temperatures in Arizona where I plan to spend a month in cold A/C and 110+ degree temperatures. It’s not exactly a sabbatical; not exactly work, but something in between.

If you can tell by my old blog entries there have been significant gaps between posts. That’s due to the lack of time between wrangling kids under 3 and wrangling a small business. Fortunately, after 9 years, the business demands less of me, and frees time to take vacation days that have been sitting like rollover minutes for the characters of Lost. This gives me some time to mull over some topics and hopefully share a few of them with you.

As we’re slowly reaching the 10th year anniversary of Geocaching and Groundspeak, it’s a good time to step back and take a look at a project that has taken just shy of a quarter of my lifetime. It’s mind-boggling to know that we’ll have a million active caches in the world by 2010.

We’re also going to try and have a heck of a celebration next year. GeoWoodstock VIII will be in our own backyard, so we hope to coordinate our celebrations with all of the new visitors to the Emerald City.

Wish me luck in the heat of Arizona. It’s a dry heat, right?

Fisher Plaza Kicks into (new) Gear

2009-07-07T16:00:00.000-07:00

This is primarily a re-posting of information sent to the customers at Internap.

Fisher Plaza has taken a good look at their infrastructure and has a pretty robust plan to ensure that there is redundancy with their electrical systems. Even though our web sites are happily chugging along, they are running on gas while a more long term solution is being worked out. This latest update outlines both the short term and long term plans to get back to normal (or super-normal, since they are adding an additional layer of protection). The plans are fluid and may change, but you can tell the facilities are going to come out stronger from this plan.

(Reposted with permission from Internap)

Near Term:

Proceeding immediately, the landlord will 1) obtain two additional generators and paralleling gear needed to provide redundancy across the 3 central risers affected by the outage. This will provide N+1 until we can return to full commercial power and allow required maintenance on the generators (required every 10 days) without any downtime. The city has approved closing the east end of John Street for the location of the additional generators. The landlord is also working with 2) Seattle City Light to get commercial power from the vault to the Life Safety System and Retail switchboard to provide a more reliable and effective solution to providing a safe work environment.

Long Term Utility Restoration:

The restoration plan involves building a completely new switchgear room outside the current vault and Switchgear Room. This would separate the Main 1 and Main 2 systems providing more reliability, new equipment, and the ability to begin construction while they are clearing out the damaged gear from the damaged Switchgear Room. All of the old Main 1 & 2 switchgear from the damaged switchgear room will be replaced with new equipment needed to provide a new Main1 service and the paralleling gear to bring up all four of the building's 1.5 MW generators on line together.

The landlord stressed that this is a very fluid plan and that suppliers are aggressively sourcing equipment, design, build, and shipping times. Once they are able to develop a specific timeline as well as Method of Operation (MOP) they will share it with Internap, and we will of course share it with our customers. They are expecting several months until full restoration.

Colotastrophe: The Day After

2009-07-04T11:11:00.000-07:00

Our servers are a bunch of primadonnas. They demand to be pampered in the greatest colocation facility in the world (if you agree with the video of Fisher Plaza touting that fact), resting on pillows of AC and fed power in Waterford crystal goblets. We literally pay more for the 5 cabinets that house the servers* than we do our entire Groundspeak office - and then some.

Around 5am Pacific today, all of our grumpy but lucid Groundspeak servers woke from their slumber to greet geocachers** who were, as one user wrote, scratching their arms in search for their next geocaching fix. Most were just happy to have the servers back online but others were asking questions about disaster recovery and communication in a crisis. Instead of finger pointing, although cathartic, I'd like to focus on what worked, what didn't, and how we can try to avert some issues if (and when) this happens again.

To set the stage, we have been hosted at Internap in the Fisher Plaza since 2002 and in that time have only had 2 significant events that related directly to facility issues. The last issue lasted around 8 hours while this one is, by far, the most signficant downtime in the history of the web site. In total we had 29 hours of downtime. Unfortunately the 29 hours were during the geocaching peak season on the busiest weekend of the year and, to compound things, a day off from work for many. The Fates were definitely conspiring to pick the worst day to bring the Geocaching.com site down.

What Worked

The usefulness of Twitter and Facebook became obvious for this crisis. All our web servers and email servers were all located at Fisher Plaza. We had very few options for posting updates, so we had to rely on outside systems to communicate with our community and our partners. I switched from Groundspeak emails to my Gmail account, and my iPhone running Tweetie helped me to get information out as I was "on the scene." By the end of the day I added an additional 800+ followers on Twitter which, in the past, was used as a toy for logging geocaching finds with my family and for the random Groundspeak update.

Also, although we didn't have the need for backups this time, we have daily backups of all our systems. Since this happened before our nightly backups occured it was close to the worst time for a data failure. At the most we would have lost a day of data. In a catastrophic event this isn't a total Fail. It just sucks.

What Didn't Work

Although I won't finger point at the cause of this issue, I will point out that Fisher Plaza people lacked any official communication with the first responders at the scene. Many clients of the building were in the dark, both figuratively and literally, while we were waiting outside for news of what really happened. Instead we had to join in on Twitter to figure out what happened. Was it a fire? (yes) Did the sprinklers turn on? (yes) OMG! Our machines are fried! (no. just the generator) If someone walked out of the building with some authority and told us what they knew - we could have passed that information on to our customers. Internap did a relatively good job at giving status updates though they were sparse and sometimes repeated. I'd give Internap a C and Fisher Plaza an F for communication.

I'll be just as hard on us and say that we should get an F for communication preparedness. Although I think we did a good job at working around our own issues with Facebook and Twitter (and this blog), we were unable to make updates available on our web pages and our iPhone application. The reason why some sites could do this and others could not is that our entire server infrastructure was in the Fisher Plaza basket. The other companies likely had better ways to switch over to a new location. Our only alternative, pointing DNS to another server, would have made it harder to get back online since many people would continue to point to the wrong machine when the servers were back with power. Since we only anticipated a ~12hr outage it made no sense to do something that could take another 24 hours to correct for some users.

What Next?

There are some obvious things to do to correct what didn't work, and some solutions that will require some thought. I'll highlight a couple of high level things we'll consider and implement.

We're not a bank, so although 29 hours is a long time to be down, we do not plan to duplicate our infrastructure so we are completely redundant. It is just too expensive to make fiscal sense. Instead, we'll ensure that in the case of a catastrophic event that we'll have the best backups and the best steps for restoring those backups to a new system. We already have a good system but we'll make it even better.

We'll have a better system for communicating with our customers, so these systems will be the focus for redundancy planning. This includes rerouting web servers and email. Even streaming my Twitter account on the front page of Geocaching.com would have been helpful for letting people know what is happening.

Lastly, we're going to create an official disaster recovery plan so everyone knows what to do at Groundspeak in the situation where there is a catastrophic event. We should always understand the worst case scenario and how to recover from it. We owe this to our customers.

For those in the US, have a Happy 4th of July! And thanks to everyone for your ongoing support of Groundspeak and the geocaching activity. From the Tweets and Facebook posts you definitely enjoy geocaching. Now go out and find a cache!

* we're not using all of the cabinets at Internap yet but we're still paying for them

** although we also run Waymarking.com and Wherigo.com, the geocaching community is easily the largest and most vocal, so I'm focusing on them for the blog. I know everyone else is just as excited to see our other sites back online.

FAIL - Fisher Plaza Outage

2009-07-03T12:29:00.000-07:00

When I get a phone call at 3am it is one of two things - either a relative has passed away or a server has gone down. Looking at those two options I would always take the latter. Bryan was on Caller ID, so I knew it was the latter. What I didn't realize was that this wasn't the ordinary check, reboot and "all's good" kind of server issue.

I tried to VPN into the machines and nothing was resolving, which is generally a Bad Thing. I dusted off my cardkey (unused for at least a year), grabbed my car keys and headed out the door in that daze that only a mom or dad knows during a 4am feeding. The usual fearful thoughts went through my head about the worst case scenarios.

Once I arrived at the Fisher Plaza building I circled the block and drove to the entrance to the parking garage, passing my keycard and receiving a satisfying beep and green light, but the gate didn't raise to let me enter. Through my dozy haze I realized that the parking garage was a dark maw with only the peek of a car in the shadows. It dawned on me that this was no longer just My Problem but Someone Else's Problem. This is one part frustrating and one part relieving. Although a power outage was a shock to the machines, we've handled this before at Fisher Plaza* and the machines were restarted with some light caresses. So I settled into the parking lot across the street and walked over to the entrance.

Arriving at the entrance I validated my initial observation that, yes, the power was out, so I struck up a conversation with some folks loitering near the entrance. "Is the power out?" I inquired, more as a conversation starter to get some additional information about what happened.

"Duh," a woman responded with a linguistic flourish. Apparently this was a KOMO employee Who Doesn't Speak with Tech Guys, but I persisted and got some general information. Yes, the power was out. It was due to a fire and one of the generators was fried. And it had been down since around 11pm. And they were here since 3 and they were sooo bored.

I realized that whatever welcome I had spent had emptied the account, so I noticed a person looking just as dazed and techie as me and struck up a conversation with him. Andrew was from PopCap Games and was also called out to check on his servers. Although most of their sites were up, their Facebook games were hosted exclusively at Internap in the Fisher Plaza Building.

After a while, more tech folks would appear and congregate, sharing any emails received by Internap which were sparse and devoid of any real content, but we'd get wisps of information from other folks who "know people" - generally the guys who actually have to get the work done. What I heard was this:

An electrical cabinet set itself on fire between 11pm and midnight
There was 6 inches of water in the parking garage where the generators are. This happened from putting out the fire.
No one was injured (thanks for asking)
One generator is fried and the other is sitting in the water so it is unusable
The Fire Marshall wasn't letting anyone in (until around 8am when I got in)
Rumors stated that the sprinklers did turn on but as alarming as it sounds, they didn't turn on in the colocation facility - so no wet machines
Kiro stations like Kiro Channel 4 in the building weren't broadcasting - instead they were broadcasting content from sister stations.

Once I got in I inspected the machines to make sure that they weren't buried in rubble or water. With nothing else I to do I headed home to wait for the "experts" to resolve the issue and update geocachers about the situation.

Realizing I couldn't get the word out on the site outages via the normal channels, like email and the web sites, I resorted to posting to Twitter, which in turn posted to Facebook. Fortunately the geocachers continued to retweet and repost the updates throughout the day, so hopefully the information is getting out there. We even updated the text for the iPhone application but that won't satisfy many folks who are experiencing network errors when trying to look up caches.

Fortunately we're in good company. Authorize.Net and Bing Travel and other larger clients are at the facility so Internap is being very serious about getting this resolved. Sometimes it is good to ride on the coattails of the bigger guys when you have a big problem to solve.

I'll update when I have more information. In the meantime I'm monitoring emails and Twitter. When the machines come back online I'll get a flurry of emails so I can head down to make sure that the machines get back online. And I'll continue to update via Twitter and Facebook until this gets resolved. You can follow the latest developments with the hashtag #fisherfire or search for Fisher Plaza or follow me @locuslingua on Twitter.

*Around 2006 Fisher Plaza lost power allegedly due to a safety switch meant to shut down power if someone was electrocuted. It's a nice safety feature unless some knucklehead hits it by mistake.

Edited: It was KOMO, Not KIRO. Thanks for the comments!

GPS Helps Predators Find your Kids

2006-04-06T10:36:00.000-07:00

The title sounds scary, huh? It makes for great headlines which is what I initially thought was the intent of Robin Raskin's syndicated column topic called "Worst Hi Tech Gifts" last November. I initially discovered the topic from John Matarese's column called "Don't waste your money" and kicked off an email letting him know of the correction regarding the now-defunct Gizmondo device:

The Gizmondo does not have a feature for predators to find your child. GPS receivers are receivers and not transmitters. I'm sure any gaming element within the unit will protect the child from broadcasting his or her location to others. The only game I'm aware of that even uses GPS is Colors and it hasn't been released yet. There is a GPS element but it is not in real time and most likely does not broadcast your child's position to others. That would be a stupid thing for a company to do.

In fact, the Gizmondo device can be set up by parents to allow them to monitor their children. So in essence it provides a feature for parents to protect their children - not the other way around.

Hope this correction helps. It's kind of damaging material for a new concept like GPS in handheld games, especially with misinformation flowing about.

Mr. Materese was kind enough to respond the same day:

I am going to refer you to robinraskin. com. She is the Computer Columnist who was the source for that story.

She and DS Simon Productions (the video feed service) offered that story to HUNDREDS of TV stations in every city in the USA. My guess is that dozens of TV stations are airing the clip saying the Gizmondo could lure sexual predators.

Since it came from a feed, and not from us, I'm not in a position to contradict what came from the feed service. You may be 100% correct....but I wish you would contact Robin Raskin.

I kicked off a similar email to Robin and never received a response. Mr. Materese also adjusted his text on his own site noted on the "Worst Hi Tech Gifts" link, much to his credit.

It's not unusual to be ignored via email so I shrugged it off and assumed it was just ignorance, or laziniess, or both that caused such a poorly penned article. Today, however, the Google news alert kicked out a newer article about the very same program. Coming from a biased newssource you have to take this with a grain of salt, but it does sound inticingly juicy:

By itself, this VNR is little more than a tri-company infomercial that plugs numerous products while trashing its competitors. And yet when laundered through credibility of TV journalism, viewers are deceived into thinking they're watching an independent news report with an impartial consumer expert.

Based on the comments regarding GPS technology in the Gizmondo device being used by sexual predators it's outright fraud, in my opinion. The damage done by articles like these sets back the widespread adoption of GPS technology. Sadly I doubt this kind of material would be picked up by the larger media organizations.

Sony Keynote Mentions Geocaching

2006-03-22T17:15:00.000-08:00

In the Playstation 3 keynote address at the Game Developer Conference in San Jose today, Phil Harrison mentioned geocaching:

Harrison also mentioned that GPS functionality will soon come on board, noting that, “Although geocaching is as underground as it gets, who can't see potential?”.

Geocaching isn't exactly an underground activity, though admittedly it has been called "the most popular outdoor activity no one knows about." There have been literally thousands of articles written about the activity, radio interviews with geocachers, and it's even listed as one of the top "treasures" on a repeating show on the Discovery Channel. So what does Harrison, the president of Sony Computer Entertainment's Worldwide Studios mean exactly?

My interpretation is that if it isn't mainstream it is underground, at least for the larger game development companies. Geocaching doesn't fit well into any of the standard genres like first person shooters, adventure games, sport or activities like golf or kayaking. It combines technology and the outdoors and is hard to put your finger on.

It does have a worldwide following but no real statistics have been reported on how popular it really is. Perhaps if that information was available it would change some minds on the subject. Or maybe it is still underground. I kind of like it that way.

Augmented Reality Gaming

2006-03-21T10:11:00.000-08:00

In the office we've been playing with names to call the new activity of Augmented Reality Gaming (or until recently, Geolocational Gaming). What came out of it was a fun term called auging or ogging. I prefer ogging though it does go a bit abstract.

What is auging? (pronounced ogging)

Auging is short for Augmented Reality Gaming. Unlike virtual reality that immerses your senses in an alternate world, augmented reality takes virtual information and overlays it on top of real space. This means instead of being in some kind of Matrix-style goo with implants stuck in your head you are using your own eyes and ears to interact in the real world. The only difference is you have additional sounds and images that you can view and interact with via headsets and/or handheld devices. No goo is necessary.

What is a typical auging experience?

There's nothing really typical about an auging experience, but the best comparison would be an adventure games in the real world. This genre started with games like Zork from Infocom and more recently through various LucasArts games like Secret of Monkey Island and Sam and Max Hit the Road. Remember Myst? Riven? These are all good examples of adventure games.

In the real world you can go on your own adventures while auging. Using a handheld device and GPS technology (which pinpoints your location) the device creates a gaming environment that reacts with your movements. For example at a zoo, If you're at the monkey cage you can "talk" to a monkey with your handheld computer because the handheld knows where you are and the game has a virtual monkey that "lives" at that location. In your handheld you may even have a virtual banana that you have in your inventory. Or, you may have to visit somewhere within the zoo to find the virtual banana that "lives" at another location. Although the monkey and banana doesn't exist in the real world you can easily imagine they do as part of your game.

What makes auging so exciting is that you are no longer chained to your home entertainment system or computer to play adventure games.

Is this Alternate Reality Gaming (ARG) like I Love Bees?

I Love Bees was an online promotion for Halo2 that involved going to phone booths in the real world to answer phone calls from in-game characters. Though there are similarities between ARGs and auging there is far less online interaction and more physical exertion involved. As popularity in auging increases the more likely that you will see the two activities blend together.

That's what I have so far.

Batteries Suck

2006-03-17T12:03:00.000-08:00

This is a half-generated thought after some frustration with a GPS enabled cell phone and its poor battery life. It also reminded me to bookmark of a recent article on battery innovation (or lack thereof).

Perhaps it goes without saying, but battery technology has lagged too far behind in comparison to computing technology. It's literally ancient. Take any electronic device made today, remove the battery, and feel the heft of the device. Now imagine a world where power can be supplied without that added bulk and weight.

GPS is one of those battery killers, though companies like Garmin are making their own hardware devices consume less and less power to determine your location. However if you tried to use a GPS on a standard GPS enabled cell phone you can expect maybe 2-3 hours on a charge. If we're going to do anything engaging with GPS and continue to make calls afterward, there will have to be some better power management features for these devices.

I expect that once GPS is actively used by more people on their cell phones that the battery consumption will be improved, but we still have to figure out a better way to power devices. We're still relying on chemical solutions which will only go so far.

Nuclear power anyone?

I'm not kidding. Nuclear power is actively being developed for use in pacemakers and other surgically implanted devices. Maybe even cell phones will use this technology some day.

Whatever the case something needs to be done about these ancient battery monstrosities. If we're going to make augmented technology useful and accessible this will be the single most important technology to improve.

Wheels of Zeus Shuts Down

2006-03-17T09:12:00.000-08:00

Today I opened up my RSS feeds to discover that the Apple co-founder Steve Wozniak closed shop on the Wheels of Zeus company before they launched a product. They did apparently license the technology to Motorola but nothing entered the marketplace. From that article I somehow made it to another article with some screenshots of the concept in action.

The concept of tracking individual objects with a localized GPS-type system is like shooting a fly with a cannon. The cost alone for the tags and central system was probably too cost prohibitive to be a reasonable product for the marketplace. I'm sure it would have gotten some legs but nothing necessarily disruptive as a technology.

What I'd like to see is Apple tackle a GPS unit design. In fact I speculate that within 5 years that Apple will get into this market.

Mologogo

2006-03-16T16:17:00.000-08:00

Today I received a Boost Moble phone I ordered from Mologogo a few days ago. It's your standard candy bar model that contains GPS and the Mologogo software preloaded. I took the extra step of installing WebJal on my desktop and using the provided USB cable to install the application on several Motorola i860 phones we had in the office.

It's great that the cable was provided with the phone - it saved me a heck of a lot of time finding the correct elements to install software on the phone.

After creating an account on Mologogo I inserted my username and password (after some issues switching from predictive text to alpha text - the winning tip is to press the page button on the phone to switch typing methods) and I was running in no less than 5 minutes with my location shown on a Google map. At least I assume it is a Google map from the similarities between the online and mobile versions.

The software has some real potential and it's a good example of a simple but inexpensive and novel use of GPS technology on a mobile phone. It's also a great solution for having a GPS unit without paying a whole lot of money for one. Hopefully more of these mobile applications will have a Goto function similar to handheld units. This feature is the only thing keeping these devices back from being competent GPS unit replacements.

Here's hoping that we'll see some improvements of this product over time. If you have $100 to dabble I'd suggest trying out their Boost phone deal to play with the novelty of GPS tracking.

PSP Future sees GPS attachment

2006-03-15T13:54:00.000-08:00

Good job Sony! The PSP will finally get that GPS attachment it always wanted:

"The GPS Receiver features an update rate of once a second and takes at most 40 seconds to get a first fix from a cold start. Developers have taken delivery of a partial set of tools for making GPS-ready software. Kutaragi mentioned a few potential applications for the device, including portable navigation software."

Yes - we can't get enough of that portable navigation software.

The first possible game is, apparently, a GPS enabled golf game where you can putt virtual balls? (just guessing). It's actually a nice idea - I can expect people driving a ball on Denny and putting in the lobby of the Fisher Building. It is one of those general GPS concepts that big companies consider when they look at GPS since it matches the industry standard of "one game to one world."

For example, GPS enabled game concepts like MogiMogi generally have random objects that are overlayed on the earth's service for you to find. Games like these can be easily generated through random placement which means anyone anywhere can drop in and play the game.

In comparison, Geocaching is a game where physical objects are placed in the world and their coordinates are shared online. By placing these physical containers there was some consideration of placement. "Is it accessible" is kind of a no-brainer when placing physical containers compared to virtual placements of objects - which by a computer can be problematic - and its lack of thoughtfulness of placement can be irritating at the least and life-threatening at the most.

The benefit, however, far outweighs the drawbacks to large computer gaming companies who are used to creating a game like Halo and knowing that it will work in the world no matter where it is played. To create an engaging GPS game though, location should be far more important. Shouldn't it be that way?

Of course! Location is key and so should be the thoughtfulness of each and every location-based game. The hardest problem will solve is making location-based games adaptable to the location without sacrificing the experience. Novelty will only go so far in this genre.

Gizmondo - Two Steps Back

2006-03-13T18:35:00.000-08:00

For those more "in the know" in the world of GPS, Gizmondo has been the most notorious company to come along in a while. We had high hopes that this little handheld device would have introduced a new evolution of gaming. For the first time it combined a GPS with a gaming platform but the company went bankrupt before it even released its first GPS enabled game called Colors.

Granted, a gun wielding gang warfare game set in the real world where you virtually fight for physical turf didn't seem like a very good idea, but at least it was a way to raise awareness of a new gaming genre.

Will this affect GPS gaming and force it two steps back? Not likely. There was so much wrong about the Gizmondo (lack of games, marketing, games, inventory and games) that they never really became known well enough to weaken the perception of location-based entertainment as a promising new industry. Fortunately the Nintendo DS and Playstation PSP were released around the same time which defeated the device soundly.

I actually have two Gizmondo devices in the office to play if there was something to play. Here's hoping that it'll go homebrew sooner than later.

Introduction

2006-03-13T17:44:00.000-08:00

Greetings.

I will not be writing about my cat.

What I will discuss is technology - but more often than not about the advancements of GPS or other location-based technologies. However I may, nay will, digress and poke fun at or otherwise vent about one technology or another. Do not be alarmed. I will eventually make it back to location-based technology.

Oh, and maybe a little about running a community-based web site.

The world speaks to us and it's intriquing to discover how technology will give it a voice. It sounds spiritual but - you'll find no druids here in this blog. Leave that to the million other blogs to write about their cats.