Trouble Maker

Trouble Maker introduces failure into your software systems intentionally.

Trouble Maker is a free open source tool for Java applications that takes down services on-demand or at scheduled intervals to test software stability.

A Case For Proactive Failure

A successful Microservices platform requires a durable and resilient environment that supports the ability to continuously deploy multiple services. Automated deployment is a must, and when possible, automated recovery from failures should be implemented, because failures will happen, (i.e. “Murphy’s Law”).

For too many reasons to count, it is nearly a guarantee that your production software systems will fail in some way.
Our answer: engineer failures into your platform’s production environment.

Instead of waiting for a failure to occur and seeing how durable and resilient your platform is, be proactive and make failure a USE CASE of your platform. If you know failures are occurring, yet pagers are not going off at 3 a.m. and the help desk is not being called, then you know your system is durable.

What Is Trouble Maker?

Trouble Maker is designed to randomly take down services during normal business hours. Additionally, it provides an ad hoc console to produce common troublesome issues in your platform so you can test its durability on demand. Trouble Maker is not dependent upon the cloud and can be used within an enterprise environment, unlike other solutions in this space based upon Amazon EC2 API.

Trouble Dashboard

The Trouble Maker Dashboard has an event log, in addition to allowing services to be selected and killed on-demand. It invokes other troublemaking issues against these services. From the dashboard, a service can be selected and the following troubles applied:

Here’s a screenshot of the Trouble Maker dashboard:

Random Kill

By default, Trouble Maker will kill a randomly-selected service once per day Monday through Friday. This option can be configured or turned off.

If you desire a stable Microservices platform, this will test its durability.

Trouble Maker is a Java Spring Boot application that communicates with a client service that has a small servlet registered with a Java API-based service application.

By default, Trouble Maker accesses Eureka to discover services, and based upon a cron task, randomly selects a service to kill (i.e. shut down). Other registries can be plugged in.

From the dashboard, a service can be selected and the following troubles applied:

• KILL – Terminate the service (i.e. system exit will be performed). Tests fail over and alert mechanisms.
• LOAD – The selected service will be invoked with numerous blocking API calls. Blocking time and number of threads can be specified. This emulates how service acts under API load.
• MEMORY – The selected service will consume memory until HEAP memory is met, then will block for a specified time. This can be used to emulate how system performs under low memory conditions.
• EXCEPTION – The selected service will throw an exception. This tests the exception handling, logging, handling, and reporting mechanisms of the service.

Please feel free to make any suggestions or submit pull requests. Our goal is for this to help organizations that are implementing Microservices to implement stable and durable platforms.