Administering Container Agents

You can create and activate agents, and configure the data collection interval.

This section covers the following key areas:

Prerequisite

Following are the prerequites to be considered before creating Kubernetes and Docker Swarm agents.

Kubernetes Agent

Each Kubernetes Agent monitors the assets inside the selected Kubernetes Service Providers. To enable the data collection, complete the following prerequisites before create agent.

  • Preparing the Kubernetes credential
  • Enabling Heapster service in monitored environment

Preparing the Kubernetes credential

The Kubernetes configuration file named KubeConfig is a standard configuration of Kubernetes and is required for Kubernetes agents to access the cluster. Foglight for Container Management verifies and supports the local Kubernetes and the following Cloud Kubernetes Service Providers. Based upon your environment, select either of approaches to get your KubeConfig file:

Data from different Kubernetes Agents with the same cluster name will be merged into one cluster.

Local Kubernetes

If you build a Kubernetes cluster locally, find this KubeConfig file under the /etc/kubernetes/admin.kubeconfig on your master node.

Azure Kubernetes Service (AKS)

Before generating the Kubernetes credentials, record the following information:

  • Azure Username
  • Azure Password
  • Azure Subscription Number
  • The name of your AKS Cluster Resource Group
  • The name of your AKS cluster

Download the Azure Command Line Interface and install it in your local platform, and then follow steps below to generate your Kubernetes credential:

  1. Run the command az login. Then a browser shows up, directing you to the Azure Portal where you should enter your Azure Username and Password to complete the authentication.
  2. Run the command: az account set –subscription
  3. Run the command: az aks get-credentials –resource-group –name
  4. Find the Kubernetes configuration file under <USER_HOME>/.kube/config on your local platform.

    The token in this Kubernetes configuration file will get expired after two years. If you don’t want the credential gets expired, refer to Foglight Container Tools for details.

Amazon Elastic Container Service for Kubernetes (EKS)

Follow the Amazon EKS offical guide Getting Started with Amazon EKS. Follow the guide and complete Create a kubeconfig for Amazon EKS in the end of the guide.

If you don’t want the credential to get expired, refer to Foglight Container Tools for details.

Google Cloud Platform Container Engine (GKE)

Download the Google Cloud Client tool and install it in your local platform, and then follow steps below to generate your Kubernetes credential.

To generate the intermediate Kubernetes credential for your cluster:

  1. Log in to your Kubernetes cluster, click Connect next to your cluster name.
  2. Copy and run the command from the Command-line access field.
  3. Find the intermediate Kubernetes configuration file under <USER_HOME>/.kube/config on your local platform. The following is the example of this intermediate Kubernetes configuration file.

    This Kubernetes configuration file cannot be used as the agent credential because the token in this file will get expired soon and “cmd-path” of the token directs to your local platform.

  4. Open Google Cloud Client tool and run the following commands to create a Kubernetes service account that grants with the cluster-admin role and the access to your Google Kubernetes Engine (GKE) cluster:
    a. kubectl create serviceaccount
    b. kubectl create clusterrolebinding –clusterrole=cluster-admin - serviceaccount=default:
    “default” in the above command is the namespace name of this service account name. You can also change to other namespace names, as needed.
    c. kubectl describe serviceaccount .
    You will get the response similar as below. Record the for later use.
    d. kubectl describe secret .
    You will get response similar as below. Record the token value (exclude “token:”) for later use.
  5. Open the intermediate Kubernetes configuration file under <USER_HOME>/.kube/config, and then add the user and change the token to the new one.

IBM Cloud Kubernetes Service

If you have created your cluster on IBM Cloud Kubernetes Service, get the access from the console as described on the Access view of the cluster. You will get a .pem file and a .yml file after performing the steps. By default IBM Cloud Kubernetes Service uses certificate authority file and token/refresh token. However, certificate authority data and service account token should be used in the Kubernetes Agent credential. After you successfully test your connection through “kubectl get nodes”, follow the steps below to generate the Kubernetes Agent credential.

  1. Run the command kubectl config view -minify=true -flatten -o json. You can view the output. Then record the for later use.
  2. Run the command kubectl create serviceaccount .
  3. Run the command kubectl describe serviceaccount . You can view the response. Then record (in this sample, it is jane-sa-token-xkqrk) for later use.
  4. Run the command kubectl describe secret . You can view the response, and then record for later use.
  5. Open the .yml file generated previously.
  6. Change the certificate authority to the data of this authority and change the users section to use <service account token>. Save your changes, and then you will get a credential file. This file will be used as the Kubernetes Agent credential to connect to your IBM cloud Kubernetes service cluster.

OpenShift Origin

If you could access the file /etc/origin/master/admin.kubeconfig on the master node, download it and use it as the Kubernetes Agent credential. If you could not access the file /etc/origin/master/admin.kubeconfig on the master node, follow the instructions below to generate a permanent credential file.

Before generating the permanent Kubernetes credentials, record the following information and ensure you have granted the privilege for accessing the cluster-wide resources:

  • OpenShift Username
  • OpenShift Password

Download the OpenShift Command Line Interface and install it in your local platform, and then follow steps below to generate your Kubernetes credential:

  1. Log in to OpenShift and generate an intermediate Kubernetes configuration file.
    a. After logging in to OpenShift, click Command Line Tools on the upper right.
    b. Click the button next to the Session token field, copy the command, and then paste it in your local Command Line Tool. Make sure to find the intermediate Kubernetes configuration file under <USER_HOME>/.kube/config on your local platform.
    c. On your local platform, browse to open this configuration file. You may see the context similar to the following. Record <config-cluster-name> for later use.
  2. The token generated in step 1 will be expired after 4 hours, however Foglight for Container Management needs a permanent Kubernetes credential. So you need to create a service account with “cluster-admin” role, and then get the authorization code (not expired) of this service account to generate our permanent Kubernetes credential.
    a. Run the command oc project .
    b. Run the command oc create serviceaccount . You can check if your service account has been created successfully using the command: kubectl get serviceaccounts.
    c. Run the command oc serviceaccounts get-token . Then you will get a token: <service-account-token>. Record this token for later use.
  3. Run the following command to grant your service account with the “cluster-admin” privilege: kubectl create clusterrolebinding – clusterrole=cluster-admin –serviceaccount=default:<service-accountname>.
  4. Generate a permanent Kubernetes configuration file and save it under <USER_HOME>/.kube/config file/credential.
    a. Open and edit the intermediate configuration file.
    b. Use kubectl to add user credentials, create new context, in the end change the existing contexts to the ones that you added in step2. For example, 
    kubectl config set-context <new-context-name> --cluster=<config-clustername> --user=<credential-name> --namespace=<project-name> 
kubectl config use-context <new-context-name>
  5. Save the current Kubernetes configuration file.

Enabling Heapster service in monitored environment

There are various approaches to enable Heapster on your Kubernetes cluster. Visit Heapster official website to determine the approach that you are going to deploy your Heapster service, or you can follow instructions in https://github.com/foglight/container to deploy your service. Some of the cloud platform Kubernetes service has enabled Heapster service for the cluster. If you have connected to the cluster, run the following command to check: kubectl cluster-info

Enabling Prometheus service in monitored environment

There are various approaches to enable Prometheus on your Kubernetes cluster. Visit Prometheus Official Website to determine the approach that you are going to deploy your Prometheus service. Or you can refer to configuration management systems, such as helm (https://github.com/helm/charts/tree/master/stable/prometheus) or ansible (https://github.com/cloudalchemy/ansible-prometheus) to simplify your installation process.

In the Foglight Container Administration dashboard, a simple template is provided for you to deploy Prometheus service onto your cluster.

Docker Swarm Agent

Each Docker Swarm Agent monitors the assets in one docker host. Docker Remote API needs to be enabled for the Docker Swarm Agent collecting data from the docker host. If TLS is enabled to secure the Docker Remote API, credential for Docker Swarm Agent needs to be prepared. Complete the following prerequisites before create agent.

Preparing Docker Swarm Agent credentials

If TLS enabled to secure Docker Remote API, then complete the following guide to get the credentials for Docker Swarm Agent for the docker host. Otherwise, continue with Enabling Docker Remote API for monitored docker host Refer to the official guide to generate the keys. Be aware that, during generating the keys, the Foglight Agent Manager host address should be in the allow access list. Docker Swarm Agent needs following credentials, you can get them when you finish the official guide.

Enabling Docker Remote API for monitored docker host

Change ExecStart in docker service startup script as below. Non-TLS secured

ExecStart=/usr/bin/dockerd –H tcp://0.0.0.0:2375 –H unix:///var/run/docker.sock

Access should be allowed to the TCP port 2375

TLS secured

If TLS enabled, complete Preparing Docker Swarm Agent credentials on page 17 first, then you will get the ca.pem, server-cert.pem and server-key.pem mentioned in the official guide. ExecStart=/usr/bin/dockerd –tlsverify –tlscacert=ca.pem –tlscert=servercert.pem –tlskey=server-key.pem -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock NOTE: Access should be allowed to the TCP port 2375

Then restart docker service.

Uploading Docker Swarm Agent credentials

If TLS is enabled to secure Docker Remote API, go through this section to upload the credential for Docker Swarm Agent. Otherwise, skip this section. When Preparing Docker Swarm Agent credentials is complete, following credentials should be generated.

  • CA Public Key
  • Client Public Key
  • Client Private Key

These are the credentials for Docker Swarm Agent, complete the following steps to upload the credentials. On the Administration > Credentials > Manage Credentials dashboard, click Add, and then select Docker CA Public Key or Docker Client Public Key or Docker Client Private Key to upload related credentials. Take Docker CA Public Key as an example.

In the Add a New “Docker CA Public Key” Credential dialog box, specify the following values:

  • Credential Properties: Click Load from file to import Docker CA Public Key, and then click Next.
  • Credential Name And Lockbox: Specify a unique name for this credential, and then click Next.
  • Resource Mapping: Click Add. In the New Resource Mapping Condition dialog box, select Target Host Name or Target Host Address for the monitored docker host
    If you select Target Host Name, then enter the host name of the monitored docker host. If you select Target Host Address, then enter the IP address of the monitored docker host.
    Click Add to finish editing New Resource Mapping Condition and back to Resource Mapping, and then click Finish.

The Docker CA Public Key has been uploaded and mapped to the docker host. To monitor this docker host, upload Docker Client Public Key and Docker Client Private Key by following the above steps.

Creating and Activating Agent

Foglight for Container Management supports Kubernetes Agent and Docker Swarm Agent.

Creating and Activating a Kubernetes Agent

To create a Kubernetes agent on a monitored host:

  1. Log in to the Foglight browser interface and make sure the left Navigation panel is open.
  2. In the navigation panel, click Container. Then the Container dashboard is displayed.
  3. In the Container dashboard, click Administration tab, and then click Create Agent. The Create Docker Agent wizard opens.
  4. Agent Manager: specify the following values, and then click Next.
    • Agent Manager: The Kubernetes Agent will be create in the selected Agent Manager.
    • Cluster Name: Customized cluster name which identifies a Kubernetes cluster.
    • Agent Name: Auto-generated agent name. You can change the name according to your requirement. It should be a unique name.
  5. Agent Properties
    • Kubernetes API Service End Point: The format is: http(s)://«url:port». If you have a Kubeconfig file, retrieve this endpoint from the file (path: clusters > cluster > server). If there are multiple clusters, find the current context related cluster server URL.
    • Collected Event Level: Set the collected event level, including ABNORMAL and ALL. ALL will collect both abnormal and normal events while Abnormal only collects abnormal events.
    • Enable Proxy: To enable the proxy, select the checkbox. Enter the Proxy Server Address and Proxy Server Port information.
    • Collector Configuration: Used to configure collection interval for inventory and metrics. You can change the collector intervals of defaultSchedule, however, this will affect all the Kubernetes Agents. Or you can create a new scheduler, configure the collector intervals, and then assign this scheduler to this agent. Collector intervals to configure:
      • inventory interval (inventory data collection interval).
      • metrics interval (performance metrics data collection interval).
  6. Credential
    • If no credentials were found for the provided resource, configure credentials:
      • Credential Properties: Click Load from file to upload the credential and click Next.
      • Credential Name and Lockbox: give a name for the credential, select a lockbox, then click Next.
    • If an existing credential is detected, go to the next step directly.
  7. Metrics Collector Currently both Prometheus metrics collector and Heapster metric collector are supported. However, users are encouraged to use the Prometheus metrics collector.
    Prometheus metrics collector-

    • Have existing Prometheus in your cluster. Ensure that you have Prometheus in your cluster before this step. We will check the service existence and health status after you click Next.

    • Select Prometheus Type.

      • Default type: allows Agent to use KubeConfig credentials to access the Prometheus service. Enter the existing Prometheus service namespace and name to configure the metrics collector
      • OpenShift Prometheus type: you need to provide OpenShift Route Hostname, Username, and Password for Agent to access the target OpenShift Prometheus.To find the Route Hostname, go to the OPENSHIFT ORIGIN dashboard, Application > Routes. Your OpenShift Login User should have the privilege to access to the Prometheus.

    • Do not have existing Prometheus in your cluster. A Prometheus Configuration wizard will appear. To configure the metrics collector, either use the default Prometheus template or upload your own Prometheus deployment .yml files through Load from files.
      Reset from template: Helps you to reset the Configure file (template) content to the default template.
      Deploy: Creates Prometheus components to your cluster with the Configure file. After clicking Deploy, a progress message will be displayed.

      • If deployed successfully, a succeeded message will be displayed. Close the Validation Result page and then click Next to finish your agent creation process.
      • If failed to deploy the Prometheus configuration, a Validation Result page will be displayed with possible solutions.

    Heapster metric collector
    Enter the existing Heapster service namespace and name to configure the metrics collector.

    Deploy the Heapster service to your cluster manually before creating an agent with Heapster as metric collector. Otherwise, connection test to your Heapster service will fail, and you cannot proceed to the next step.

  8. Summary: click Finish.
  9. Then, the agent will be created and activated automatically.

Creating and Activating a Docker Swarm Agent

Each Docker Swarm Agent monitored one docker host. If the docker host belongs to a Docker Swarm cluster, it will be considered as a manager/worker node. Otherwise, it will be considered to be a standalone docker host.

For a Docker Swarm cluster, you should create one Docker Swarm Agent for one host in the cluster, and if you want to monitor the whole cluster environment, you need to create all the Docker Swarm Agents for all the hosts in the cluster.

To create a Docker Swarm agent on a monitored host:

  1. Login in to the Foglight browser interface and make sure the left navigation panel is open.
  2. In the navigation panel, click Administration > Agents > Agent Status. The Agent Status dashboard opens.
  3. In the Agent Status dashboard, click Create Agent. The Create Agent wizard opens.
  4. Host Selector: Select the monitored host that you want to monitor with the Docker Swarm agent instance that you are about to create, and then click Next.

    In order to select the host, the Foglight Agent Manager must be installed and running on the monitored host.

  5. Agent Type and Instance Name: Specify the following values, and then click Next.

    • Agent Type: Select DockerSwarmAgent from the agent type list.

    • Agent Name: Specify the name of the agent instance that you are about to create using either of the following approaches:

    • Generic Name: This option is selected by default. A generic name is a combination of the host name and the agent type and uses the following syntax: agent_type@host_name.

    • Specify Name: Type that name in the Name field. For example, MyAgent.

  6. On the Summary page, review the choices you have made, and then click Finish. The Agents table refreshes automatically, showing the new Docker Swarm Agent.
  7. On the Agents table, select the Docker Swarm Agent that you create, click Edit Properties, and then click Modify the private properties for this agent.
  8. In the Agents properties view, check if the following values have been configured based upon your environment:
    • Name: give a name to the monitored docker host, it should be unique.
    • Host Name: IP address or host name of the monitored docker host.
    • Docker Remote API End Point: Docker Remote API endpoint of the monitored docker host. For more information, see Enabling Docker Remote API for monitored docker host.
    • Swarm Name: specify the swarm cluster name for display. If the swarm name is kept as “default”, then the cluster name will be displayed as “default (cluster ID)” on the dashboard. If a customized name is input here, then the customized cluster name will be displayed on the dashboard.

      Ensure that the docker host inside the same cluster has the same configuration for Swarm Name.

  9. Return back to the Agents table, select the above property changed Docker Swarm Agent, and then click Activate. The new Docker Swarm Agent is created and data will be shown on the Monitoring tab after a few minutes.

Configuring data collection interval

The default data collection interval of agents is set to 5 minutes by default. Foglight for Container Management enables you to change this collection interval as needed.

Changing the data collection interval will take effect for all Kubernetes agents and Docker Swarm agents.

To configure the data collection interval:

  1. In the navigation panel, select Administration > Agents > Agent Status.
  2. On the Agent Status dashboard, select the Kubernetes agent that you use to monitoring the container environment, and then click Edit Properties.
  3. In the Edit Properties dashboard, click Edit next to the Collector Config field.
  4. In the KubernetesAgent or DockerSwarmAgent Collector Config dialog box, change the following values, as needed:
    • Inventory Collector: Specifies the interval for collecting components.
    • Metrics Collector: Specifies the interval for collecting metrics.
  5. Click Save.