Batch Architecture - Designing Your Threadpools - Part 2

In the last article we discussed the setup of a cluster. Once the cluster is setup, the next step is to design and configure your threadpools. Before I illustrate how to quickly configure your threadpools here are a few things to understand about threadpools:

• Threadpools are Long Running JVMs. The idea behind threadpools are they are long running JVMS that accept work (from submitters) and process that work. Each individual instance of a threadpool is an individual running JVM on a host (or hosts).
• Threadpools are Named. Each threadpool is named with a tag (known as the Threadpool Name). This tag is used when running a batch process to target specific JVM's to perform the processing. The names are up to the individual implementation.
• Threadpools Can Have Multiple Instances. Threadpools can be singular instance or have multiple instances within a host or across hosts.
• Threadpools have thread limits. Each instance of a threadpool has a physical thread limit. This is not the Java thread limit but the maximum number of threads that can be safely executed in the instance.
• Threadpools with the same name have cumulative limits when clustered. When multiple instances of the same thread pool name are available, the number of threads available is the sum total of all instances. This is regardless whether the instances are on the same host or across host as long as they are in the same cluster.

A summary of the above is shown in the figure below:

For the above scenarios:

Scenario Option Example Pool Comments ASingle Thread Pool on a Single HostPOOL1 This is the most simplest scenario. BMulti Thread Pool Instances on a Single HostPOOL2 The number of threads across this scenario are cumulative. In this example there are ten (10) threads available. CMulti Thread Pool on Multi HostsPOOL3

This is a clustered setup across hosts. Again the threads available are cumulative and in this case there are twelve (12) threads available.

Note: The second instance of POOL3 can have different thread limits to reflect the capacity on the host. In most cases, the number of threads will be the same but it is possible to change the configuration on a host to reflect the capacity of that individual machine.

Note: You can combine any of these scenarios for a complex batch architecture.

Building Your Threadpool Configuration

As with the Cluster configuration the best way of building your threadpool configuration is using the Batch Edit (bedit.sh) utility. There are two commands available to you:

• The bedit.sh -w command is recommended as your initial command to create a set of default thread configurations in threadpoolworker.properties file. This file is used by the threadpoolworker.sh command by default.
• The bedit.sh -l <arg> command is used to create custom threadpools with the <arg> used to denote the template to base the configuration against. The product ships a cache and job template and it is possible to create customer templates directly from the command. The utility generates a threadpool.<arg>.properties file. To use the template use the threadpoolworker.sh -l <arg> command line.

Here are some guidelines when building the threadpool configuration:

• Some Settings Are At The Template Level. Some of the settings are common to all threadpools using the template. For example, JVM Settings, Caching and Internal Threads are specified at  the template level.
• Create An Appropriate List Of Threadpool Names And Thread Limits In The File. The list of threadpools can be configured with names and limits.
• Use Cache Template for Cache Threadpools. In a complex architecture with lots of threadpool instances, it is recommended to invest in creating a cache threadpool to optimize network traffic.
• Use Job Template for Focussed Threadpools. If optimization of JVM settings, caching etc is required use the job template or create a custom template to configure these settings.

The bedit.sh utility allows for configuration of settings from the templates. For example:

$ bedit.sh -w

Batch Configuration Editor [threadpoolworker.properties]
--------------------------------------------------------------------------
Current Settings
  minheap (1024m)
  maxheap (1024m)
  daemon (false)
  rmiport (7540)
  dkidisabled (false)
  storage (true)
  distthds (4)
  invocthds (4)
  role (OUAF_TPW)
  jmxstartport (7540)
  l2 (READ_WRITE)
  devmode (false)
  ollogdir (/u02/sploutput/demo)
  ollogretain ()
  thdlogretain ()
  timed_thdlog_dis (false)
  pool.1
      poolname (POOL1)
      threads (10)
  pool.2
      poolname (POOL2)
      threads (5)
  pool.3
      poolname (POOL3)
      threads (7)
>   

 

Use the help command on each setting for advice.

When designing your threadpools there are several guidelines that can apply:

• Simple Is Best. One of the key capabilities of the architect is that you have access to a full range of alternative configuration to suit your needs. To quote a famous movie "With Great Power Comes Great Responsibility", so it is recommended not to go overboard with the complexity of the configuration. For example, in non-Production environment use a small number of threadpools so keep it simple. When designing your threadpool architecture, balance maintenance over technical elegance. The more complex solutions can increase maintenance costs so keep the solution as simple as you can.
• Consider Specialist Threadpools for L2 Caching. Some of the ancillary processes in the product require the L2 Cache to be disabled. This is because they are updating the information and the cache will actually adversely affect the performance of these processes. Processes such as Configuration Migration Assistant, LDAP Import and some of the conversion processes require the caching to be disabled. In this case create a template where a dedicated threadpool that turns off L2 Caching.
• Threadpools Need Only To Be Running As Needed. One misnomer about threadpools is that they must be up ALL the time to operate. This is not true. They only need to be active when a batch processes needs access to the particular threadpool. When they are not being used, any threadpool JVM's are still consuming memory and CPU (even just a little). There is a fundamental principal in Information Technology which is "Thou Shalt Not Waste Resources". An idle threadpool is still consuming resources, including memory, that can be best used by other active processes.
• Consider Specialist Threadpools for Critical Processing. A threadpool will accept any work from any process it is targeted for. Whilst this is flexible, it can cause issues with critical resources. When a critical process is executed in your architecture, it is best to make sure there are resources available to process it efficiently. If other resources are sharing the same threadpools then those critical processes are competing for resources from lesser critical processes. One technique is to setup dedicated threadpools, with optimized settings, for exclusive use of critical processes.
• There Are Limits. Even though it is possible to run many threadpools in your architecture there are limits to consider. The most obvious is memory. Each threadpool instance is a running JVM reserving memory for use by threads. By the default, this is between 768MB - 1GB (or more) per JVM. Your physical memory may be the limit that decides how many active JVM's are possible on a particular host (do not forget the operating system also needs some memory as well). Another limit will be contention on resources such as records and disk. One technique that has proven useful is to monitor throughput (records per second) and to increase threading or threadpools until this starts to reduce.

The above techniques are but a few that are useful in designing your threadpools. The next step in the process is to decide the submitters and the number of threads to consider across these threadpools. This will be subject of the next part of the series.