Lokomotive Equinix Metal configuration reference

    Introduction

    This configuration reference provides information on configuring a Lokomotive cluster on Equinix Metal, formerly Packet, with all the configuration options available to the user.

    Prerequisites

    Configuration

    To create a Lokomotive cluster, we need to define a configuration.

    Example configuration file:

    # mycluster.lokocfg
    variable "equinixmetal_token" {}
    variable "asset_dir" {}
    variable "facility" {}
    variable "cluster_name" {}
    variable "controllers_count" {}
    variable "workers_count" {}
    variable "controller_type" {}
    variable "controller_clc_snippets" {}
    variable "workers_type" {}
    variable "dns_zone" {}
    variable "route53_zone_id" {}
    variable "equinixmetal_project_id" {}
    variable "ssh_public_keys" {}
    variable "management_cidrs" {}
    variable "node_private_cidrs" {}
    variable "state_s3_bucket" {}
    variable "lock_dynamodb_table" {}
    variable "oidc_issuer_url" {}
    variable "oidc_client_id" {}
    variable "oidc_username_claim" {}
    variable "oidc_groups_claim" {}
    variable "worker_clc_snippets" {}
    variable "worker_pool_facility" {}
    
    backend "s3" {
      bucket         = var.state_s3_bucket
      key            = var.state_s3_key
      region         = var.state_s3_region
      dynamodb_table = var.lock_dynamodb_table
    }
    
    # backend "local" {
    #   path = "path/to/local/file"
    #}
    
    cluster "equinixmetal" {
      auth_token = var.equinixmetal_token
    
      asset_dir = var.asset_dir
    
      cluster_name = var.cluster_name
    
      controller_count = var.controllers_count
    
      controller_type = "c3.small.x86"
    
      controller_clc_snippets = var.controller_clc.snippets
    
      facility = var.facility
    
      os_channel = "stable"
    
      os_version = "current"
    
      os_arch = "amd64"
    
      ipxe_script_url = ""
    
      project_id = var.equinixmetal_project_id
    
      dns {
        zone     = var.dns_zone
        provider = "route53"
      }
    
      ssh_pubkeys = var.ssh_public_keys
    
      management_cidrs = var.management_cidrs
    
      node_private_cidrs = var.node_private_cidrs
    
      cluster_domain_suffix = "cluster.local"
    
      network_mtu = 1500
    
      tags {
        key1 = "value1"
        key2 = "value2"
      }
    
      enable_aggregation = true
    
      enable_tls_bootstrap = true
    
      encrypt_pod_traffic = true
    
      enable_reporting = false
    
      network_ip_autodetection_method = "first-found"
    
      pod_cidr = "10.2.0.0/16"
    
      service_cidr = "10.3.0.0/16"
    
      reservation_ids = { controller-0 = "55555f20-a1fb-55bd-1e11-11af11d11111" }
    
      reservation_ids_default = ""
    
      certs_validity_period_hours = 8760
    
      conntrack_max_per_core = 32768
    
      oidc {
        issuer_url     = var.oidc_issuer_url
        client_id      = var.oidc_client_id
        username_claim = var.oidc_username_claim
        groups_claim   = var.oidc_groups_claim
      }
    
      worker_pool "worker-pool-1" {
        count = var.workers_count
    
        clc_snippets = var.worker_clc_snippets
    
        tags = {
          pool = "storage"
        }
    
        ipxe_script_url = ""
    
        os_arch = "amd64"
    
        disable_bgp = false
    
        facility = var.worker_pool_facility
    
        node_type = var.workers_type
    
        os_channel = "stable"
    
        os_version = "current"
    
        labels = {
          "testlabel" = ""
        }
    
        taints = {
          "nodeType" = "storage:NoSchedule"
        }
    
        setup_raid = false
    
        setup_raid_hdd = false
    
        setup_raid_ssd = false
    
        setup_raid_ssd_fs = false
      }
    }
    

    NOTE: Should you feel differently about the default values, you can set default values using the variable block in the cluster configuration.

    Example:

    The default for node_type is c3.small.x86. If you wish to change the default, then you define the variable and use it to refer in the cluster configuration.

    variable "custom_default_worker_type" {
      default = "c2.medium.x86"
    }
    
    .
    .
    .
    node_type = var.custom_default_worker_type
    .
    .
    .
    
    

    Attribute reference

    Argument Description Default Type Required
    auth_token Equinix Metal Project API token. Can be supplied using PACKET_AUTH_TOKEN environment variable. - string false
    asset_dir Location where Lokomotive stores cluster assets. - string true
    cluster_name Name of the cluster. - string true
    tags List of tags that will be propagated to master nodes. - map(string) false
    controller_count Number of controller nodes. 1 number false
    controller_type Equinix Metal instance type for controllers. Changing this field on existing cluster will be ignored. To actually apply it, cluster needs to be destroyed and re-created. “c3.small.x86” string false
    controller_clc_snippets Controller Flatcar Container Linux Config snippets. [] list(string) false
    dns DNS configuration block. - object true
    dns.zone A DNS zone to use for the cluster. The following format is used for cluster-related DNS records: <record>.<cluster_name>.<dns_zone> - string true
    dns.provider DNS provider to use for the cluster. Valid values: cloudflare, route53, manual. - string true
    oidc OIDC configuration block. - object false
    oidc.issuer_url URL of the provider which allows the API server to discover public signing keys. Only URLs which use the https:// scheme are accepted. - string false
    oidc.client_id A client id that all tokens must be issued for. “clusterauth” string false
    oidc.username_claim JWT claim to use as the user name. “email” string false
    oidc.groups_claim JWT claim to use as the user’s group. “groups” string false
    facility Equinix Metal facility to use for deploying the cluster. - string false
    project_id Equinix Metal project ID. - string true
    ssh_pubkeys List of SSH public keys for user core. Each element must be specified in a valid OpenSSH public key format, as defined in RFC 4253 Section 6.6, e.g. “ssh-rsa AAAAB3N…”. - list(string) true
    os_arch Flatcar Container Linux architecture to install (amd64, arm64). “amd64” string false
    os_channel Flatcar Container Linux channel to install from (stable, beta, alpha, edge). “stable” string false
    os_version Flatcar Container Linux version to install. Version such as “2303.3.1” or “current”. “current” string false
    ipxe_script_url Boot via iPXE. Required for arm64. - string false
    management_cidrs List of IPv4 CIDRs authorized to access or manage the cluster. Example [“0.0.0.0/0”] to allow all. - list(string) true
    node_private_cidr (Deprecated, use node_private_cidrs instead) Private IPv4 CIDR of the nodes used to allow inter-node traffic. Example “10.0.0.0/8”. - string true
    node_private_cidrs List of Private IPv4 CIDRs of the nodes used to allow inter-node traffic. Example [“10.0.0.0/8”]. - list(string) true
    enable_aggregation Enable the Kubernetes Aggregation Layer. true bool false
    enable_tls_bootstrap Enable TLS bootstraping for Kubelet. true bool false
    encrypt_pod_traffic Enable in-cluster pod traffic encryption. If true network_mtu is reduced by 60 to make room for the encryption header. false bool false
    ignore_x509_cn_check Ignore check of common name in x509 certificates. If any application is built pre golang 1.15 then API server rejects x509 from such application, enable this to get around apiserver. false bool false
    network_mtu Physical Network MTU. 1500 number false
    pod_cidr CIDR IPv4 range to assign Kubernetes pods. “10.2.0.0/16” string false
    service_cidr CIDR IPv4 range to assign Kubernetes services. “10.3.0.0/16” string false
    cluster_domain_suffix Cluster’s DNS domain. “cluster.local” string false
    enable_reporting Enables usage or analytics reporting to upstream. false bool false
    reservation_ids Block with Equinix Metal hardware reservation IDs for controller nodes. Each key must have the format controller-${index} and the value is the reservation UUID. Can’t be combined with reservation_ids_default. Key indexes must be sequential and start from 0. Example: reservation_ids = { controller-0 = "<reservation_id>" }. - map(string) false
    reservation_ids_default Default reservation ID for controllers. The valuenext-available will choose any reservation that matches the pool’s device type and facility. Can’t be combined with reservation_ids - string false
    certs_validity_period_hours Validity of all the certificates in hours. 8760 number false
    conntrack_max_per_core Maximum number of entries in conntrack table per CPU on all nodes in the cluster. If you require more fain-grained control over this value, set it to 0 and add CLC snippet setting `net.netfilter.nf_conntrack_max sysctl setting per node pool. See [Flatcar documentation about sysctl] for more details. 32768 number false
    worker_pool Configuration block for worker pools. There can be more than one. - list(object) true
    worker_pool.count Number of workers in the worker pool. Can be changed afterwards to add or delete workers. 1 number true
    worker_pool.clc_snippets Flatcar Container Linux Config snippets for nodes in the worker pool. [] list(string) false
    worker_pool.tags List of tags that will be propagated to nodes in the worker pool. - map(string) false
    worker_pool.cpu_manager_policy CPU Manager policy to use. Possible values: none, static. “none” string false
    worker_pool.disable_bgp Disable BGP on nodes. Nodes won’t be able to connect to Equinix Metal BGP peers. false bool false
    worker_pool.ipxe_script_url Boot via iPXE. Required for arm64. - string false
    worker_pool.os_arch Flatcar Container Linux architecture to install (amd64, arm64). “amd64” string false
    worker_pool.os_channel Flatcar Container Linux channel to install from (stable, beta, alpha, edge). “stable” string false
    worker_pool.os_version Flatcar Container Linux version to install. Version such as “2303.3.1” or “current”. “current” string false
    worker_pool.node_type Equinix Metal instance type for worker nodes. “c3.small.x86” string false
    worker_pool.facility Equinix Metal facility to use for deploying the worker pool. Enable “Backend Transfer” on the Equinix Metal project for this to work. Same as controller nodes. string false
    worker_pool.labels Map of extra Kubernetes Node labels for worker nodes. - map(string) false
    worker_pool.taints Map of Taints to assign to worker nodes. - map(string) false
    worker_pool.reservation_ids Block with Equinix Metal’s hardware reservation IDs for worker nodes. Each key must have the format worker-${index} and the value is the reservation UUID. Can’t be combined with reservation_ids_default. Key indexes must be sequential and start from 0. Example: reservation_ids = { worker-0 = "<reservation_id>" }. - map(string) false
    worker_pool.reservation_ids_default Default reservation ID for workers. The valuenext-available will choose any reservation that matches the pool’s device type and facility. Can’t be combined with reservation_ids. - string false
    worker_pool.setup_raid Attempt to create a RAID 0 from extra disks to be used for persistent container storage. Can’t be used with setup_raid_hdd nor setup_raid_sdd. false bool false
    worker_pool.setup_raid_hdd Attempt to create a RAID 0 from extra Hard Disk drives only, to be used for persistent container storage. Can’t be used with setup_raid nor setup_raid_sdd. false bool false
    worker_pool.setup_raid_ssd Attempt to create a RAID 0 from extra Solid State Drives only, to be used for persistent container storage. Can’t be used with setup_raid nor setup_raid_hdd. false bool false
    worker_pool.setup_raid_ssd_fs When set to true file system will be created on SSD RAID device and will be mounted on /mnt/node-local-ssd-storage. To use the raw device set it to false. false bool false

    Applying

    To create the cluster, execute the following command:

    lokoctl cluster apply
    

    Destroying

    To destroy the Lokomotive cluster, execute the following command:

    lokoctl cluster destroy --confirm
    

    ARM support and hybrid clusters

    Lokomotive and Flatcar currently include an alpha-quality preview for the Equinix Metal arm64 server type c2.large.arm. They can be used for both worker and controller nodes. Besides specifying them in controller_type/node_type you need to configure some additional variables in the respective controller/worker module:

    os_arch = "arm64"
    os_channel = "alpha"
    ipxe_script_url = "http://alpha.release.flatcar-linux.net/arm64-usr/current/flatcar_production_packet.ipxe"
    

    The iPXE boot variable can be removed once Flatcar is available for installation in the Equinix Metal OS menu for the ARM servers.

    If you have a hybrid cluster with both x86 and ARM nodes, you need to either use Docker multiarch images such as the standard debian:latest or python:latest images, or restrict Pods to nodes of the correct architecture with an entry like this for ARM (or with amd64 for x86) in your YAML deployment:

    nodeSelector:
      kubernetes.io/arch: arm64
    

    An example on how to build multiarch images yourself is here .

    Note: Lokomotive Components are not supported in this preview, so if you need them you’ll have to install them manually using Kubernetes directly.