tag:blogger.com,1999:blog-24022931.post4713730692227983425..comments2013-04-22T05:39:05.324-07:00Comments on Locus Lingua - Jeremy Irish's Blog: Colotastrophe: The Day AfterJeremy Irishhttp://www.blogger.com/profile/09436728562365583820noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-24022931.post-20487144707379366902009-07-07T15:54:40.256-07:002009-07-07T15:54:40.256-07:00This wasn't your failure. Just keep this great...This wasn't your failure. Just keep this great idea running. Today I renewed my premium account ;)<br /><br />Cheers ;) Jan aka rheingauerUnknownhttps://www.blogger.com/profile/12093389177978430145noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-37363743897907016272009-07-05T19:50:44.250-07:002009-07-05T19:50:44.250-07:00Jeremy,
Thanks for letting us know what happened....Jeremy,<br /><br />Thanks for letting us know what happened. Also, thanks for doing all you can to make geocaching enjoyable for all. Some things, while forseeable, are not always preventable.<br /><br />HAPPY GEOCACHING!!!flame-redhttps://www.blogger.com/profile/00633215469548091583noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-52289657012605576172009-07-05T02:28:58.029-07:002009-07-05T02:28:58.029-07:00Hi Jeremy,
Many thanks for all your very good job ...Hi Jeremy,<br />Many thanks for all your very good job during this difficult moment. I would give you an A for communication.<br />Regards and Happy GeocachingUnknownhttps://www.blogger.com/profile/07607759360092025714noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-26767232388795907702009-07-05T00:22:45.940-07:002009-07-05T00:22:45.940-07:00Just to say thanks to everyone involved for gettin...Just to say thanks to everyone involved for getting things up-and-running so quickly. I wasn't a huge fan of Twitter before this happened, although I had an account, but this event has made me a fan! Thanks for all your updates, Jeremy, it was great to be kept informed as to what was occuring, and I was able to keep friends who aren't twitterers in the loop, too.<br /><br />Thanks again!geotrowelhttps://www.blogger.com/profile/07919419598617318973noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-43953945731477648962009-07-04T22:59:00.654-07:002009-07-04T22:59:00.654-07:00That is the way to go. Learn from the situation an...That is the way to go. Learn from the situation and the mistakes made, take one on the chin, fix it and move on.<br />Glad to hear that all ended well.AntonD69https://www.blogger.com/profile/16176154885805058841noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-50927665121331974442009-07-04T21:48:40.470-07:002009-07-04T21:48:40.470-07:00It goes without saying that those of us that were ...It goes without saying that those of us that were networking your twitter and facebook updates throughout the night with other geocaching group sites helped spread the word to the starving fun-seekers out there. Thanks for your firm's dedication and like mtn-man said....take a few days off...you all deserve it!Hockeyhickhttps://www.blogger.com/profile/16947403353853535147noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-68484020768379268102009-07-04T21:08:37.791-07:002009-07-04T21:08:37.791-07:00Jeremy,
As far as I'm concerned you did what...Jeremy, <br /><br />As far as I'm concerned you did what needed to be done. You can always 'what if' yourself to death on DR exercises and still miss something. You've learned some valuable lessons. I never really saw the need for Twitter before but was one of the +800 who started following you just to know the status. I can see how it can really help in these times. Again, you've learned something, no one was hurt and the world didn't end. <br /><br />You and your team all did a great job. I trust that you've got some things to think about.Anonymoushttps://www.blogger.com/profile/00915385183265797544noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-87355560804822356692009-07-04T19:28:24.911-07:002009-07-04T19:28:24.911-07:00I will be the first to say that I really appreciat...I will be the first to say that I really appreciate all that Jeremy and his wonderful staff do for us. That being said, even though some of us feel like we would die without Geocaching, it isn't a life or death situation. I'd say that since I started in 2003, Groundspeak's UP time is phenomenal! I am sure there are some cachers out there that feel Geocaching needs to install a fully robust, immediately switching, all the bells and whistles, super-duper back-up system. But I know I survived without it for 29 hours. I don't think suicide ever popped into my head one single time.<br /><br />Thanks again Geocaching staff!Chadhttps://www.blogger.com/profile/14329678134055575573noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-60942284326314154272009-07-04T17:24:03.834-07:002009-07-04T17:24:03.834-07:00Thanks for the excellent recap and honest assessme...Thanks for the excellent recap and honest assessment. I am sorry that you and Elias lost your holiday. I like FrankBroughton's idea of a separate blog and posting the weekly email and information that is posted in the announcements section of the forums. When people get used to it, it would be a great location for an emergency update.<br /><br />Take a day off next week guys! If it makes you feel any better, I am working this entire weekend and did not get home last night until 2:40 AM. Same tomorrow.mtn-manhttps://www.blogger.com/profile/14309478929530735200noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-2005474042063304692009-07-04T16:26:16.515-07:002009-07-04T16:26:16.515-07:00Jeremy,
I have an idea for 'ya to mull over. ...Jeremy,<br /><br />I have an idea for 'ya to mull over. The hosting company I use for my sites has a blog setup that is in a different Data Center than its main equipment that hosts its forums and client section. This blog among other things is used in emergency downtime. Only used once that I recall as now their forum and other items are on failover.<br /><br />Groundspeak could do this same thing and use this blog as an avenue for announcing site updates and such. Use the blog instead of the thread in the forum for the site update announcements. The weekly newsletter could even be posted on the blog. Along with other things. Keep it simple so you do not have to have it hosted on a dedicated server.<br /><br />A link to the blog could even be setup from the current forum thread. vBulletin allows a thread to be a link, I suppose IPB does too.<br /><br />Customers would get used to checking the blog and thus it would be perfect for future emergencies (hopefully never needed, but in reality it will). <br /><br />Just priming the pump with ideas.<br /><br />Thanks for the yeoman job of keeping us updated yesterday. You did very well. I grade you an A. <br /><br />Frank BroughtonFrank Broughtonhttps://www.blogger.com/profile/10104028853021212041noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-86929228742819193462009-07-04T15:23:17.651-07:002009-07-04T15:23:17.651-07:00Absoblogginlutely! You'll need to re-read wha...Absoblogginlutely! You'll need to re-read what Jeremy said. "Since this happened before our nightly backups occured it was close to the worst time for a data failure."<br /><br />That means the disaster happened at the worst possible time. Not the backups. You can't schedule disasters. Which means you can't schedule a backup around it. The disaster happening at the worst possible time does not mean there was a better time to schedule the backup.Unknownhttps://www.blogger.com/profile/15500272378246525675noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-67123430355751474662009-07-04T15:13:41.562-07:002009-07-04T15:13:41.562-07:00Hi,
I agree that a dns entry with a short ttl, say...Hi,<br />I agree that a dns entry with a short ttl, say 15 mins to a web server hosting a sorry page would have been better. Also you might consider an entry for the api record so that apps would return something meaningful too. I do highly available it infrastructure for a living so i'd be happy to provide free advice to you guys about what your options are and the associated costs.Jeremyhttps://www.blogger.com/profile/02105429998826609108noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-50857076721054203672009-07-04T15:12:13.595-07:002009-07-04T15:12:13.595-07:00Although I commiserate with the unfortunate timing...Although I commiserate with the unfortunate timing I'd just like to represent the masses and say I knew nothing at all of what was going on apart from it being evident a major issue had occurred.<br />As a professional DBA I am surprised that you don't backup transaction logs on a periodic basis allowing point-in-time recovery. Maybe something you could look into?<br />I was certainly very glad I maintain an off-line database.<br />Hope you were able enjoyed the holiday to some extent.<br />Bob.Petehttps://www.blogger.com/profile/13509974199291726561noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-91112123624417448822009-07-04T14:40:17.755-07:002009-07-04T14:40:17.755-07:00Good Job. I am lucky to read it her so far Away in...Good Job. I am lucky to read it her so far Away in BerlinJens L. Hessehttps://www.blogger.com/profile/05965674145401351111noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-61211155081374969322009-07-04T14:37:34.725-07:002009-07-04T14:37:34.725-07:00Thanks for this, Jeremy...I'm one of your 800+...Thanks for this, Jeremy...I'm one of your 800+ new followers on Twitter. At the end of the day this was not a life-and-death situation, but the fact that you and Elias were on the scene PDQ demonstrates your professionalism.<br /><br />We're just about to take our annual vacation, and my wife will be furious that I can now Twitter on my mobile...the PQ will be set to run the night before ;)<br /><br />geoid: 83192Mittellegihttps://www.blogger.com/profile/07396310595614212139noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-87613017337088691922009-07-04T14:28:44.476-07:002009-07-04T14:28:44.476-07:00A lot of us are not that informed on all that goes...A lot of us are not that informed on all that goes into maintaining gc.com. As a simple webmaster of a small geocaching association, from the begining I was impressed with how you and Elias did everything you could do to let everyone know why they couldn't access the site.<br /><br />I knew there'd be those without iphones and other media access that would not have any idea why their busiest geocaching day was going down the tubes. I stayed with it all night and kept posting to our website an on-the-fly account of what was happening. I couldn't have done it if you had not stayed with us on Twitter, Jeremy.<br /><br />On a site that usually gets maybe 50 hits on a given day, suddenly myscsg.com was getting thousands. Other companies affected by the outtage were posting a link to the SCSG site on Twitter. As far as I know, there was nowhere else to go to get a coherent chronolgy of why this was happening. I knew I had suddenly found myself and our site needed! I stayed with it all night posting your twits and any other info I could gather, and was genuinely impressed with how you used what you had to get the word out.<br /><br />I know I can say with confidence that you and Elias did more communicating to your customers with less available equipment than this laymen could have ever dreamed possible, and I know I can speak for all the simple geocachers out here when I say we are pleased and impressed with your efforts! Ya'll saved the holiday! You guys rock!waypointazoidhttps://www.blogger.com/profile/13082046313172945531noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-90384908137094593952009-07-04T14:14:41.233-07:002009-07-04T14:14:41.233-07:00I have to say that you did one of the best jobs I ...I have to say that you did one of the best jobs I have ever seen in providing updates to the community. It goes a long way toward keeping us all calm.Anonymoushttps://www.blogger.com/profile/11554854449775184139noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-79622493565473759712009-07-04T13:36:53.366-07:002009-07-04T13:36:53.366-07:00My .02 is that even though you may not have been h...My .02 is that even though you may not have been happy with Groundspeaks communication to it's customers, as a customer of GC.com and Groundspeak I was more than happy with the continuing coverage by the GC frontman. Nice job all of you.Seegarshttps://www.blogger.com/profile/09005247167847368882noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-19582131998914660662009-07-04T13:22:36.088-07:002009-07-04T13:22:36.088-07:00Well, the good news is, that once you have DR plan...Well, the good news is, that once you have DR plan, you're most likely not going to need it:)<br /><br />Furthermore, I'd like to point out that the fisher plaza as a collocation center sucks big time. Which genius decided to use sprinklers in geni's rooms/server rooms? Don't know what the housing prices are in US, but down here in EU I can effort quite nice geographical cluster for just simple webhosting I'm running as a side job (although I don't have 7 racks full of servers:)<br /><br />Anyway, I'm glad that the gc.com is back online, all the other details are now irrelevant...Pavelhttps://www.blogger.com/profile/06896629903543692436noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-9353661913399016712009-07-04T13:20:01.369-07:002009-07-04T13:20:01.369-07:00Jeremy,
Thanks for the update on what happened an...Jeremy,<br /><br />Thanks for the update on what happened and your plans for any future events.. I definitely understand the not going for 100% redundency due to the costs.. The facebook updates worked very well for me and it sounded like you had everything in place you could when given the okay with power and A/C..<br /><br />I hope your servers are protected by halon and not water.. That expense is well worth it..<br /><br />Keep up the good work Mr. Pres / co-owner.. Your customers really do appreciate it..<br /><br />Dennis B<br />(Burgi Dad)burgi dadhttps://www.blogger.com/profile/17180134153650279281noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-70781389672800037302009-07-04T13:15:00.950-07:002009-07-04T13:15:00.950-07:00Jeremy,
Thank you for all that you have done to k...Jeremy,<br /><br />Thank you for all that you have done to keep those affected by the Fisher Plaza fire updated. You and others on twitter were our life line. I am in Florida. My hosting company (dotster.com) is evidently in the Plaza and was shut down, thus shutting my web site and e-mails down. Yet, the only way I could discover what was happening to them, and my web sites was you, and Twitter.<br /><br />I guess I am now a full fan of Twitter.<br /><br />Your blog also helped.<br /><br />ThanksBillhttps://www.blogger.com/profile/00103208311809108676noreply@blogger.comtag:blogger.com,1999:blog-24022931.post-69970622657290375182009-07-04T12:59:39.374-07:002009-07-04T12:59:39.374-07:00thanks for all your hard work this weekend. Thankf...thanks for all your hard work this weekend. Thankfully my pocket queries were sent just before the fire ;-)<br />It raised some interesting questions, a lot of which you seem to have answered and also raised some questions in my mind about how the company I work for and my clients would react in a similar situation.<br /><br />You mentioned that your backups occured at the worst possible time during the day - are you going to take steps to change this happening - can you afford to lose a days worth of data? I bet there would be some pretty mad people if a days worth of logs, photos and tracks were deleted.<br /><br />As to the web rerouting - couldn't you set a short ttl on the dns and point to a blog or twitter account that would tell people how to refresh their dns once the servers were back up?Absoblogginlutely!https://www.blogger.com/profile/10375412476973524867noreply@blogger.com