ok, so today was a mess.
lets see.. two days ago one of our big customer's reporting just locked. fuck. it followed a power outage, so it was partially written off to chance. phew.
then yesterday, it happened again. fuck. rebooted the box, no clue. see timeout errors from mod_proxy. then it happened again. fuck. rebooted. sent email to reporting vendor asking if there were changes in 4.0 -> 4.1.10 .. we had to upgrade to fix other problems in generating pdf reports.
they write back asking for a java thread dump. okay. get to work today, customer is locked. fuck. grabbed a thread dump. deadlock. fuck. see that it is in the cache code for the reporting code. can i disable cache? nope. fuck. email thread dump to vendor. since they are in germany, and got this at 10am, it was too close to the end of their day so no reply. fuck. lockups continued to happen a few more times.
started working on code to allow the reload of webapps in sevak. you can't use the standard admin servlet in jetty because it won't re-use the special WebApplicationContext that interfaces with the Avalon lifecycle.
so i got the code mostly working, but i need to reset the context classloader when being called from the mx4j. then i had the idea! version 4.0 worked fine. lets just run two different versions in different classloaders. yea! fuck. both servlets are in the same webapp. can't do classloader isolation at the servlet level inside a webapp. mod_rewrite to the rescue! lets rewrite all pdf requests to a new webapp since its single-connection oriented, version the callback required for the javabean viewer with the other method.
finally got all that working after enough ant-fu. uploaded to customer, and since their sysadmin is out of town, the backup never verified.. so more of the same deadlocks tomorrow. whee.
then yesterday, it happened again. fuck. rebooted the box, no clue. see timeout errors from mod_proxy. then it happened again. fuck. rebooted. sent email to reporting vendor asking if there were changes in 4.0 -> 4.1.10 .. we had to upgrade to fix other problems in generating pdf reports.
they write back asking for a java thread dump. okay. get to work today, customer is locked. fuck. grabbed a thread dump. deadlock. fuck. see that it is in the cache code for the reporting code. can i disable cache? nope. fuck. email thread dump to vendor. since they are in germany, and got this at 10am, it was too close to the end of their day so no reply. fuck. lockups continued to happen a few more times.
started working on code to allow the reload of webapps in sevak. you can't use the standard admin servlet in jetty because it won't re-use the special WebApplicationContext that interfaces with the Avalon lifecycle.
so i got the code mostly working, but i need to reset the context classloader when being called from the mx4j. then i had the idea! version 4.0 worked fine. lets just run two different versions in different classloaders. yea! fuck. both servlets are in the same webapp. can't do classloader isolation at the servlet level inside a webapp. mod_rewrite to the rescue! lets rewrite all pdf requests to a new webapp since its single-connection oriented, version the callback required for the javabean viewer with the other method.
finally got all that working after enough ant-fu. uploaded to customer, and since their sysadmin is out of town, the backup never verified.. so more of the same deadlocks tomorrow. whee.