Release notes¶

Version 1.59¶

Helm chart version 1.59.x

Mysql migration¶

Background migrations, running during the core container startup, may take up to tens of minutes. Use the Helm --timeout 30m option to adjust the timeout accordingly.

If the ODM service is unable to start within the allocated time, increase the failureThreshold value for the core container.

Removal of Application Container Persistent Volume¶

We have removed the Persistent Volume Claim (PVC) and the necessary configuration for the application's container. This removal is safe, so there's no need for concern.

Version 1.58¶

Danger

This version must be installed before proceeding with the next update.

Helm chart version 1.58.3

Clickhouse migration¶

In current release we moved from standalone Clickhouse container to the Clickhouse cluster that will be controlled by Altinity clickhouse operator.
We automated process of transferring data from the standalone version to the cluster version of clickhouse, migration will be executed during upgrade process.

Things that you have keep in mind before upgrading process:

Migration time depends on resources(mostly on CPU and Disk IO) allocated for Clickhouse instances, during our tests we've mentioned average speed 50Gb per hour for instances with 4CPU/16Gb RAM
It's necessary not to set flags as --wait and --timeout during upgrading process because of migration time
The new ClickHouse cluster must have 25% more disk space than the standalone variant.
Optional: We've developed a tool that checks consistency of data, you could use it before and after the upgrading process and compare results of those checks just to be sure that everything went as expected

Following steps¶

(Upgrade flow) Proceed with the odm installation, take itnto account:

a. odm-ops chart will install the Altinity ClickHouse operator with pre-configured settings.

b. In your custom values for odm chart adjust parameters for ClickHouse and Altinity ClickHouse (requests, limits, disk size +25%, etc.). We suggest to temporary increase resources for both of Clickhouses since it will reduce timing for migration.

c. VERY IMPORTANT! Make sure that old clickhouse is NOT disabled! Path in values is clickhouse.enabled. By default, it's enabled.

d. DO NOT apply recommendations.yaml file from example as is, it's recommendation ONLY for new installations!

e. Resources path in values for old clickhouse is clickhouse.resources for the new one altinity.clickhouse.installation.spec.templates.podTemplate.spec.mainContainer.resources

f. Persistance size path for old clickhouse is clickhouse.persistence.size for the new one altinity.clickhouse.installation.spec.templates.volumeClaimTemplate.spec.resources.requests.storage
A job named odm-clickhouse-helper will appear in Kubernetes, and it will handle the migration.

a. During the ClickHouse migration, ODM will continue to operate, but all writes to ClickHouse will be queued.

b. Wait until the odm-clickhouse-helper job completes, indicating that the migration is done.
Disable clickhouse and clickhouseHelper in Helm values. You can refer to the example disable-old-clickhouse-after-upgrade.yaml.
Update ODM one last time with helm upgrade .... This will disable the old ClickHouse.

Helm examples changes¶

New examples for different ODM configuration options have been added to the examples helm chart directory, and all old ones have been updated.
Additionally, recommendations for computing resources have been included.

Helm configuration changes¶

From this release, we are using fully original Docker images for the OSS components of ODM. It is not recommended to update them independently.

From:

mysql:
  image:
    registry: 091468197733.dkr.ecr.us-east-1.amazonaws.com
    repository: genestack/mysql

mailcatcher:
  image:
    registry: 091468197733.dkr.ecr.us-east-1.amazonaws.com
    repository: genestack/mailcatcher

clickhouse:
  image:
    registry: 091468197733.dkr.ecr.us-east-1.amazonaws.com
    repository: genestack/clickhouse

nginx:
  image:
    registry: 091468197733.dkr.ecr.us-east-1.amazonaws.com
    repository: genestack/nginx

To:

mysql:
  image:
    registry: docker.io
    repository: mysql

mailcatcher:
  image:
    registry: docker.io
    repository: dockage/mailcatcher

clickhouse:
  image:
    registry: docker.io
    repository: clickhouse/clickhouse-server

nginx:
  image:
    registry: docker.io
    repository: nginxinc/nginx-unprivileged

Now you can mount any file with any content into any container in ODM! For example, your certificates. This feature required adding the full path in all existing ODM configuration files.

From:

core:
  configurationFiles:
    "application.yaml":

applications:
  configurationFiles:
    "application.yaml":
    "microsoft.openid.ini":
    "okta.openid.ini":
    "google.openid.ini":

mysql:
  configurationFiles:
    "genestack.cnf":

funcFile:
  configurationFiles:
    "application.yaml":

funcJob:
  configurationFiles:
    "application.yaml":

linkService:
  configurationFiles:
    "application.yaml":

clickhouse:
  configurationFiles:
    "config.yaml":
    "users.yaml":

nginx:
  configurationFiles:
    "odm.conf":
    "proxy-pass-parameters.conf":

To:

core:
  files:
    "/var/lib/genestack/properties/application.yaml":

applications:
  files:
    "/var/lib/genestack/properties/application.yaml":
    "/var/lib/genestack/properties/microsoft.openid.ini":
    "/var/lib/genestack/properties/okta.openid.ini":
    "/var/lib/genestack/properties/google.openid.ini":

mysql:
  files:
    "/etc/mysql/conf.d/genestack.cnf":

funcFile:
  files:
    "/app/config/application.yaml":

funcJob:
  files:
    "/app/config/application.yaml":

linkService:
  files:
    "/app/config/application.yaml":

clickhouse:
  files:
    "/etc/clickhouse-server/config.d/config.yaml":
    "/etc/clickhouse-server/users.d/users.yaml":

nginx:
  files:
    "/etc/nginx/conf.d/odm.conf":
    "/etc/nginx/conf.d/proxy-pass-parameters.conf":

The AWS credentials for connecting to S3 in core and applications have been removed. If you have these parameters, you can safely delete them.

core:
  files:
    "/var/lib/genestack/properties/application.yaml":
      backend:
        aws:
          region: ""
          endpoint:
            url: ""
          access:
            key: ""
          secret:
            key: ""

Danger

Important! The AWS region in the application must remain! You can delete only the endpoint, access and secret parameters.

applications:
  files:
    "/var/lib/genestack/properties/application.yaml":
      frontend:
        aws:
          region: "{{ .Values.credentials.awsS3Region }}"
          endpoint:
            url: ""
          access:
            key: ""
          secret:
            key: ""

Configuration file settings.py.local has been removed. If you are using it, you can safely delete it.
```
core:
  files:
    "settings.py.local":
```

The previously added BusyBox image for ClickHouse has been removed. If you are using it, you can safely delete it.

clickhouse:
  busyboxImage:
    registry: docker.io
    repository: busybox
    tag: 1.36.1

Version 1.57¶

Danger

This version must be installed before proceeding with the next update.

Helm chart version 1.57.0

Helm configuration changes¶

Removed the link to the database for the service func-file. If you have it in your values.yaml, then you can safely remove the spring map completely.

funcFile:
  configurationFiles:
    "application.yaml":
      spring:
        datasource:
          # -- Mysql jdbc URL
          url: "jdbc:mysql://..."

For the Clickhouse busybox image, the ability to set the repository and version has been added.

clickhouse:
  busyboxImage:
    # -- Image registry
    registry: docker.io
    # -- Image repository
    repository: busybox
    # -- Image tag
    tag: 1.36.1

Version 1.56¶

Helm chart version 1.56.1

Export metrics to Genestack¶

Fluent-bit was introduced as an extra service tasked with collecting and dispatching metrics in Prometheus format to a Genestack.

These metrics encompass technical and/or product-related data, devoid of any sensitive information.

If you wish to deactivate this functionality, you can do so by configuring the following parameter:

fluent-bit:
  enabled: false

Helm configuration changes¶

Now organization name and hostname are in a global section:

From:

odmFrontendHostname: odm.local
applications:
  configurationFiles:
    "application.yaml":
      frontend:
        ui:
          organization:
            name: "Genestack"

To:

global:
  hostname: odm.local
  organizationName: "Genestack"

Version 1.55¶

Helm chart version 1.55.4

Configure ODM usage together with encrypted S3 bucket (SSE-KMS and SSE-S3 only)¶

Introduction¶

You can find configuration examples in the ODM Helm chart.

In case you have several AWS credentials in your configuration, you need to modify only the credentials for accessing the bucket in specified as frontend.aws.bucket.

SSE-KMS¶

To enable uploading into an SSE-KMS encrypted bucket, you need to customize func-file configuration. The following configuration example uses a bucket encrypted by SSE-KMS with the name <BUCKET_NAME>. The bucket configuration should specify the algorithm aws:kms as preferredAlgorithm. Additionally, the property kmsCmkId should be added with a value equal to key id arn:aws:kms:... if the bucket policy requires this key to be explicitly send on PUT request. The func-file section in the configuration should look like this:

SSE-S3¶

The SSE-S3 encryption type is default to the most buckets. To force ODM request this type of encryption from S3 provider for <BUCKET_NAME>, you need to specify the preferredAlgorithm property with the value AES256:

On `storage_config` section configuration in `func-file`¶

Keep in mind that func-file reads the storage_config section sequentially. You can create specific configurations for individual buckets, e.g., if one has SSE-KMS encryption while others do not. To do this, as the first item in the list, you'll need to specify the bucket with the specific configuration and its name. Then, provide the general configuration for the other buckets using the wildcard symbol *. ODM will only upload files to the bucket, specified as frontend.aws.bucket property, regardless to storage_config section.

Genestack pod separation¶

Example on the image section, but it's applicable for sections with backend/frontend separation.

ApplicationSettings changes showed separately:

From:

genestack:
  image:
    backend:
      registry: 091468197733.dkr.ecr.us-east-1.amazonaws.com
      repository: genestack/core
      pullPolicy: Always
      pullSecrets: []
    frontend:
      registry: 091468197733.dkr.ecr.us-east-1.amazonaws.com
      repository: genestack/applications
      pullPolicy: Always
      pullSecrets: []

To:

core:
  image:
    registry: 091468197733.dkr.ecr.us-east-1.amazonaws.com
    repository: genestack/core
    pullPolicy: Always
    pullSecrets: []

applications:
  image:
    registry: 091468197733.dkr.ecr.us-east-1.amazonaws.com
    repository: genestack/applications
    pullPolicy: Always
    pullSecrets: []

Application settings rework¶

From:

genestack:
  applicationSettings:
    backend:
      properties:
        # backend.properties file content
      propertiesAuth:
        # backend-credentials.properties file content
      propertiesLimits:
        # limits.yaml file content
      predefinedSystemUsers:
        # token and password for technical odm users
      predefinedUsers:
        # predefined-users.json file content
    frontend:
      properties:
        # frontend.properties file content
        "google.openid.ini":
          # google.openid.ini file content
        "microsoft.openid.ini":
          # microsoft.openid.ini file content
        "okta.openid.ini":
          # okta.openid.ini file content
      propertiesAuth:
        # frontend-credentials.properties file content
      monitoringThresholds:
        # monitoring-thresholds.yaml file content
    saml:
      # saml directory content

To:

core:
  configurationFiles:
    "application.yaml":
      # backend.properties and backend-credentials.properties files content in YAML format
    "settings.py.local":
      # settings.py.local file content
  secretFiles:
    # saml directory content

applications:
  configurationFiles:
    "application.yaml":
    # frontend.properties and frontend-credentials.properties files content in YAML format
    "google.openid.ini":
    # google.openid.ini file content
    "microsoft.openid.ini":
    # microsoft.openid.ini file content
    "okta.openid.ini":
    # okta.openid.ini file content

High-level paths renaming in values.yaml¶

Solr¶

From:

index: {}  # Solr configuration

To:

solr: {}  # Solr configuration

Clickhouse¶

From:

txIndex: {}  # Clickhouse configuration

To:

clickhouse: {}  # Clickhouse configuration

Mysql¶

From:

db: {}  # Mysql configuration

To:

mysql: {}  # Mysql configuration

Nginx¶

From:

proxy: {}  # Nginx configuration

To:

nginx: {}  # Nginx configuration

Release notes¶

Version 1.59¶

Mysql migration¶

Removal of Application Container Persistent Volume¶

Version 1.58¶

Clickhouse migration¶

Following steps¶

Helm examples changes¶

Helm configuration changes¶

Version 1.57¶

Helm configuration changes¶

Version 1.56¶

Export metrics to Genestack¶

Helm configuration changes¶

Version 1.55¶

Configure ODM usage together with encrypted S3 bucket (SSE-KMS and SSE-S3 only)¶

Introduction¶

SSE-KMS¶

SSE-S3¶

On storage_config section configuration in func-file¶

Genestack pod separation¶

Application settings rework¶

High-level paths renaming in values.yaml¶

Solr¶

Clickhouse¶

Mysql¶

Nginx¶

On `storage_config` section configuration in `func-file`¶