What's New in RapidMiner AI Hub 9.10.4?

Released: February 7th 2022

Upgrading to RapidMiner AI Hub 9.10.4 includes database and RapidMiner AI Hub home directory migration steps. Ensure that you do not abort AI Hub startup while migration is in progress!

Before upgrading, the following steps are crucial and should be followed:

In your running AI Hub instance, temporarily pause all Schedules
Verify that all jobs are in a final state like finished, error, stopped or timed out
- Consider force stopping all currently running or pending jobs depending on your needs or
- Wait for all jobs to be executed
- Cross-check execution state on the Executions web page of AI Hub
Shutdown all Job Agents attached to this AI Hub instance
Shutdown AI Hub
Backup (instructions depend on your setup)
Upgrade to AI Hub to 9.10.4 (instructions depend on your setup)
Start AI Hub and wait for all migrations to finish
Observe startup and migration progress by tracking logs which are located inside the RapidMiner AI Hub home directory under $rmHomeDir/logs/ folder, e.g. migration.log, migration-eb.log
Once everything succeeded, start your Job Agents and wait for them to show up in the Queues page of AI Hub
Resume all Schedules which you’ve temporarily paused before

If you’ve accidentally upgraded although not all executions have been finished before upgrading (see instructions above), non-final executions like pending or running ones might show up in the Job Archive view of the Executions page.

Please head over to the Troubleshooting section outlining more details regarding “Job Archive contains pending or running jobs”.

Enhancements

Added Job Archive and improved periodic Job Cleanup mechanism to archive and clean archived jobs more efficiently
- Added database migration for moving existing job related tables to archive tables prefixed with A_:
  - jobservice_job_context_out renamed to a_jobservice_job_context_out
  - jobservice_job_context_in renamed to a_jobservice_job_context_in
  - jobservice_job_context_macro renamed to a_jobservice_job_context_macro
  - jobservice_job_context renamed to a_jobservice_job_context
  - jobservice_job_error renamed to a_jobservice_job_error
  - jobservice_job_log renamed to a_jobservice_job_log
  - jobservice_operator_progress renamed to a_jobservice_operator_progress
  - jobservice_job renamed to a_jobservice_job
  - (Re-)Creation of the following tables is handled automatically after renaming migration succeeded:
    - jobservice_job_context_out
    - jobservice_job_context_in
    - jobservice_job_context_macro
    - jobservice_job_context
    - jobservice_job_error
    - jobservice_job_log
    - jobservice_operator_progress
    - jobservice_job
    - Unique and foreign constraints now have proper identifiers
- Added a migration step for Job Cleanup
  - Changed how cleanup is enabled to jobservice.scheduled.archive.jobCleanup.enabled = true, before the cleanup was enabled by setting the jobservice.scheduled.jobCleanup.maxAge property
  - Changed existing property jobservice.scheduled.jobCleanup.cronExpression to jobservice.scheduled.archive.jobCleanup.jobCronExpression
  - Added new property jobservice.scheduled.archive.jobCleanup.jobContextCronExpression to clean up the job context of deleted jobs separately
  - Added new property jobservice.scheduled.archive.jobCleanup.jobBatchSize to set the number of jobs to be deleted at once
  - Added new property jobservice.scheduled.archive.jobCleanup.jobContextBatchSize to set the number of job contexts to be deleted at once
Added mechanism to log System.out and System.err inside the Job Container
- Property is jobcontainer.systemOutLog.enabled (false by default)
- Enable by setting property jobagent.container.jvmCustomOptions=-Djobcontainer.systemOutLog.enabled=true inside the agent.properties file
Added more logging when Job Agents retrieve a job deletion message via broker
Added size and checksum mismatch checks for LFS object uploading which can be used in addition to the LFS /verify POST endpoint
- Change repositories.lfsEnableUploadSizeCheck to enable/disable the size check for LFS PUT endpoint (defaults to true)
- Change repositories.lfsEnableUploadChecksumCheck to enable/disable the checksum check for LFS PUT endpoint (defaults to true)
Added mechanism to remove unsuccessful LFS uploads directly after failure (repositories.lfsRemoveUnsuccessfulUploads, defaults to true)
Added a migration step which checks LFS objects consistency
- Migration step is warning only and will not prevent startup of AI Hub
- Overall migration result is written into the log file migration-eb.log
- In addition, there are per Project $projectId-consistency-result.json files inside the $rmHomeDir/data/repositories/git_lfs_server folder with more information
Increased robustness of Job Agent execution status propagation to AI Hub
- Each job has a result.json file inside the respective $jaHome/data/jobs/jobId folder
- The file is used to determine the job state, if the executing Job Container has been flagged as unreachable due to high load or other system/environment conditions
- If creating the result.json file fails, a result.error file is created inside the directory of that job
Introduced cleanup mechanism for completed and propagated (sent via broker to AI Hub) job state events inside the Job Agent (false by default)
- Enable via jobagent.jobStateEventsCleanup.enabled = true inside the agent.properties file
- Adjust interval in which state events are deleted via jobagent.jobStateEventsCleanup.interval = 120000 inside the agent.properties file, defaults to 60000 milliseconds
Global theme color changes
Added a safety net for CVE-2021-44228
Bump radoop-proxy to 1.2.3
Bump integrated JDBC driver version postgresql to 42.3.2

Categories

Versions

What's New in RapidMiner AI Hub 9.10.4?

Enhancements