Concurrent Reload Settings in Qlik Sense Enterprise

The max concurrent reloads can be configured in the Qlik Sense Management Console.

1. Open the?Qlik Sense Management Console
2. Navigate to?Schedulers?
3. Enable the?Advanced?section (if not already on by default)
4. Set the required?max concurrent reloads
   ?

   

   ?

?

IMPORTANT TO REMEMBER

• A reload node could run?NumberOfCores - 2
• If for example, the reload node has 8 CPU Cores then it could possibly run 6 concurrent reloads at its maximum.
• A reload task typically consumes at least one core, sometimes more
• With more concurrent tasks running than there are number of cores, the total time to complete a number of tasks that is larger than the number of cores does not improve with more parallel tasks running due to starvation of the server
• Each reload running is loaded into RAM. This means that the average RAM footprint grows with more parallel reloads
• With more parallel reloads than cores the probability for total starvation increases since the QIX engine is CPU hungry and other processes (for example, the operating system) might not get enough CPU cycles to perform their duties
• Use the guideline,?NumberOfCPUCores - 2?in performing capacity planning on the possible amount of concurrent scheduled reloads.
• Scheduled reloads are queued, however, if not enough resources in the system
• To validate that environment needs proper capacity planning, go to?C:\ProgramData\Qlik\Sense\Log\Scheduler\Trace\<Server>_System_Scheduler.txt, an example log shows below:
  ?

  <ServerName>_System_Scheduler.txt
  Domain\qvservice ? ?Engine connection released. 5 of 4 used?
  Domain\qvservice?? ?Engine connection 6 of 4 established?
  Domain\qvservice?? ?Request for engine-connection dequeued. Total in queue: 25

Concurrent settings

Use the "Max concurrent reloads" to limit the maximal concurrent tasks can be run at same time on?current node. By default, it's set to 4, which means only 4 tasks can be run at same time on this node.
When the 5th task comes in:

1. It will be queued by sequence.?
2. The queue has a time setting, which will eliminate (erase) a queued task if the timeout limit reaches. By default, it's set to 30 mins, in this case, if none of the first 4 tasks finishes in 30 minutes after the 5th task comes in, then the 5th task will be cancelled due to timeout. It has to be triggered again (manually or next scheduled time slot).
3. Once one of the 4 running tasks finishes, the 5th task will be executed on this node?if it's not timed out.
4. If you set timeout to 0, this will make the queue never timeout.
5. It's suggested not to run more than 10 concurrent reloads at once, unless the schedular node has an extreme amount of resources dedicated to it.
6. If there is low memory or cores available on the node, then some tasks may kick off however take several hours to complete. Adding resources to the node running the reloads is suggested.

Multi-node deployment

On a multi-node deployment, tasks will be balanced from the manager node to any node(s) designated as workers.

It's highly advised to check if the central node configured is set to?Manager?and?Worker?or?Manager. When set to?Manager, it will send all reload jobs to the reload/scheduler nodes, as it should. However if a central node is set to?Manager and Worker, this means the Central node will also be involved in performing reloads. This is not recommended.?

The work flow looks as follows:

1. Manager node receives a new task execution request.
2. Manager node checks the resource availability on each of the worker nodes.?
3. Manager node assigns this task to?the?node with the lowest number of running tasks per?"Max concurrent reloads" setting.

The improvement to track the?Max concurrent reloads?can, if desired, be disabled. This reverts Sense to an older load balancing method that relies only on CPU usage.?

To disable the setting:

1. Open the?Scheduler.exe.config, which by default is located in:?C:\Program?Files\Qlik\Sense\Scheduler\Scheduler.exe.config
2. Set?DisableLegacyLoadBalancingBehavior?setting to false
3. Restart Qlik Sense Scheduler Service
4. Repeat these actions on each node of the cluster running the Qlik Sense Scheduler Service

Example

In our example, we allow?one?concurrent reload, but we assume that?two?reloads are executed at the same time.

• If Task A is executed first, Task B is queued.
• If Task A is executed first and then fails, Task B is executed after the failure.
• After Task A's failure, Task A is queued and will execute after Task B has finished.