Payment Notification Problems Today - Feb 17, 2010
Ahhh, notifications, one of our favorite subjects. I do want to limit the scope of this entry to just the PAYMENT RECEIVED notifications that went out earlier this afternoon. I know that a sizable chunk of us are still having various problems receiving notifications on a timely basis and we are working on that*.
For those of us that unexpectedly received "payment received" emails/notifications for v-gifts, paid accounts or other items from the LiveJournal store, please keep in mind:
We apologize for the confusion we've caused and hope this answered the most common concerns and questions.
We rewrote the way our web and worker servers interface with Memcached, which required upgrading all of the Perl Storable modules from version 2.12 to 2.15. We had to do all of them BEFORE the next patch release, and do them pretty much simultaneously since the 2.12 and 2.15 versions are incompatible as far as being able to read information that was written in the previous version.
For those of us familiar with desktop programs like a word processor, think how sometimes the latest updated program will automatically convert a file that you saved in a previous version and you can read it fine, but if you edit it and then try to save it, there will be a warning saying the old version of the program won't be able to read the new file? Does this even make sense to anyone or am I rambling again? It's the same concept with the Perl modules that we upgraded today.
The upgrade was supposed to be seamless and no one would even notice, except for the good things that were to come later.
However, in the process of upgrading, we discovered that 2 of our latest servers somehow had an even newer, 2.21 version on there! So anything they've been WRITING, has been unable to be read and acted upon with the old version. And then we upgraded to 2.15 that can actually read the jobs and well... some of those jobs that disappeared? Surprise, we found them!
It was actually a non-pleasant surprise to us as well as you.
* We've actually started over-hauling our notification system -- the actual code -- to make it more robust and cohesive, as well as give us better debugging so we can more accurately track down the source of problems. This rewrite started in the last release but that release also had it's share of problems. We are continuing to release fixes and hope to fix the largest problems very soon.
For those of us that unexpectedly received "payment received" emails/notifications for v-gifts, paid accounts or other items from the LiveJournal store, please keep in mind:
- These are just the notifications, we have not charged you for them again even though they were sent today.
- Many of these notifications are either duplicates or were never originally sent when you made the purchase weeks or months ago.
- The notification does not mean you were enrolled in some auto-renew program without your consent.
We apologize for the confusion we've caused and hope this answered the most common concerns and questions.
We rewrote the way our web and worker servers interface with Memcached, which required upgrading all of the Perl Storable modules from version 2.12 to 2.15. We had to do all of them BEFORE the next patch release, and do them pretty much simultaneously since the 2.12 and 2.15 versions are incompatible as far as being able to read information that was written in the previous version.
For those of us familiar with desktop programs like a word processor, think how sometimes the latest updated program will automatically convert a file that you saved in a previous version and you can read it fine, but if you edit it and then try to save it, there will be a warning saying the old version of the program won't be able to read the new file? Does this even make sense to anyone or am I rambling again? It's the same concept with the Perl modules that we upgraded today.
The upgrade was supposed to be seamless and no one would even notice, except for the good things that were to come later.
However, in the process of upgrading, we discovered that 2 of our latest servers somehow had an even newer, 2.21 version on there! So anything they've been WRITING, has been unable to be read and acted upon with the old version. And then we upgraded to 2.15 that can actually read the jobs and well... some of those jobs that disappeared? Surprise, we found them!
It was actually a non-pleasant surprise to us as well as you.
* We've actually started over-hauling our notification system -- the actual code -- to make it more robust and cohesive, as well as give us better debugging so we can more accurately track down the source of problems. This rewrite started in the last release but that release also had it's share of problems. We are continuing to release fixes and hope to fix the largest problems very soon.