Queue management
One of the most powerful features of Rapidminer Server is that it can execute RapidMiner processes either immediately or as scheduled processes. Processes can be scheduled to be executed once at a specific point in time or periodically at specific intervals.
Rapidminer Server utilizes a queueing infrastructure where processes are submitted when they are ready for execution. The Job Agent picks up processes once they are made available in the queue and executes them.
Note: A queue can have one or several Job Agents connected to it. Each of these connected Job Agents can execute an arbitrary number of RapidMiner processes concurrently. However, the default value for parallel execution is set to three.
Queues
RapidMiner Server is shipped with a pre-configured queue named DEFAULT, but it is possible for the user to create additional queues.
Note: All Job Agents initialized during the installation of RapidMiner server will automatically connect to the DEFAULT queue.
To view this list of queues go to Processes > Queues located on the left side of the page:
This will redirect you to the Queues page where each configured queue is listed along with it's status information:
The table below explains the status information associated with a Queue
| Field | Description |
|---|---|
| Running jobs | the number of processes that are currently being executed |
| Pending jobs | the number of processes awaiting execution |
| Permitted groups | a list of all the groups that are allowed to submit processes to the queue for execution |
| Job Agents | the number of Job Agents connected to the queue |
Adding a Queue
To configure a new queue, navigate to Processes > Queues. Click on Add Queue button located at the top right side of the queues page:
The Add Queue dialog will appear, where you can enter the name of the queue.
After you have entered the queue's name, click the Create button to complete the process. The new queue should be immediately visible in the Queues page.
Removing a Queue
To remove an existing queue, navigate to Process > Queues. This will bring you to the Queues page.
Click on the delete button next to the queue name you want to delete.
Note: The DEFAULT queue cannot be deleted.
Managing a Queue's permissions
A queue can be configured to allow process execution to specific groups. To change these permissions navigate to Process > Queues. This will bring you to the Queues interface.
Click the Edit link next to the Permitted groups property of the queue.
This will bring up the Manage permissions dialog:
To permit a group to execute processes from the queue, select the group name from the left hand side group list and click > (Add) arrow. This action will move the selected group(s) to the group list on the right hand side.
To restrict a group from executing processes from the queue, select the group from the group list on the right hand side and click < (Remove) arrow. This action will move the selected group(s) to the group list on the left hand side.
When done click Save.
Connecting a Job Agent to an existing Queue
To install and connect to a Job Agent an existing queue, navigate to Processes > Queues. This will bring you to the Queues interface.
Click the Install link next to the Job Agents property of the queue.
You will be prompted to download a pre-configured Job Agent (in ZIP format) on to your local disk. Once the download is completed, unzip the archive.
To start the Job Agent, navigate to the unzipped Job Agent directory and run in the command line the /bin/rapidminer-jobagent.bat file in case you are using a windows operating system, or /bin/rapidminer-jobagent.sh file in case you are using other operating systems. This will boot up the Job agent which will automatically connect to the queue for which it is configured.
Purging a Queue's pending executions
The queue supports to purge all pending executions, for example when a large amount of jobs have been accidentally submitted by a schedule.
The purge operation is destructive. There’s no way to get already purged jobs back into queue besides re-submitting them.
When pending executions exist in the queue, click on the Purge button and confirm the action. Depending on the number of pending executions, the operation might take several minutes to hours. Once purging is complete, the purge progress indicator will disappear from the responding queue action and all pending executes have been marked as stopped.
