Shutdown application context
Recent technical issue at work involves how a sprint-boot application shutdown itself. There are several types of applications we have over here, some are simply spring-boot web applications or scheduled applications, which should never shutdown. Some are SQS listeners that also do not shutdown. On the other hand, there are batch type of applications that should shutdown gracefully. And such batch type might be SQS listeners too. So someone built a SQS handler that can shutdown when after a given time there is no message, and another guy built an override handler that also shutdown when a message matches some criteria (it does not matter if message is handled, it seems nobody cares as the message will be retried in the next scheduled batch). And some batch simply shutdown after it is done. The technical issue was that, they refuse to shutdown. It turns out such application has defined DataSource bean using the commons-dbcp2 module. In there there is a thread pool that has to be destroyed by calling BasicDataSource.close(), which is automatically done by spring-boot if everything is configured as intended. But in one case, the DataSource is not defined as a bean, so its "close()" method is not called. The IDE was able to catch some errors but the stack trace was wrong - did not show the problematic DataSource but point to a different one. In another case, DataSource has JMX warnings after application shutdown, so instead of disabling JMX, developer added (destroyMethod="") to the bean annotation, which means the "close()" method is ignored by spring-boot. Finally, some application does not close its application context. It is fine to ignore the "close()" in most cases, as most applications are supposed to run forever. And if the context is closed for a web application, such application will shutdown immediately after startup. Having subtle difference in an often boilerplate "Main" class is not a good practice when there are so many modules to maintain.. and it caused problem. If developer has to call System.exit(0) that is a bad smell.
Previously there were also issues with thread pool executors and executor services that do not shutdown, but luckily they are easy to discover and fix. So the tiny issue of application shutdown is almost over.
In the works there are still challenging questions as how to submit tasks so logically it is easy to understand, code is easy to maintain; and publisher can be suppressed so that when consumer is slow, there won't be OOM; and publisher can retry and record errors if consumer failed at any point. A common problematic pattern is to submit task without taking care of errors, so firstly exceptions got propagated to caller at "join()" when caller does not care about that; secondly tasks that caller is yet to submit will be lost after the incident. It is another nasty situation as different user may have different requirement. Most times if one sub-task failed, caller should abort asap. But when sub-tasks are independent, as in the case of lambda handling kinesis messages, caller has to make sure no message is lost. Why there has to be so many things to take care of in programmer's world? I cannot find a good analog but maybe if you have 10 cups and 5 of them must be used for beer and others for milk or beverage but absolutely no alcohol and you have to maintain the markers forever even if they got into dish washer and microwaves occasionally and have to be identified later and separately handled?
Frankly when the entire world is talking about Spark on EMR, it is not a good idea to spend hours and hours to fix this kind of technical issues. With System.exit(0) it worked reliably in production, or with missing messages it is ok to fix by simply replaying the missing updates, at the same time creating jobs in the operations half of developer's work. Technical issues help create the generations of computer workers, because they create work and limit competing with new generations. So, thanks "Programming using Google search" or "Get things done with StackOverflow", so much burden freed with them