## Introduction Karmada, which stands for "Kubernetes Armada," is an open-source, multi-cloud, and multi-cluster Kubernetes orchestration system. It allows you to run cloud-native applications across multiple Kubernetes clusters and regions. It has advanced scheduling capabilities, high availability, centralized multi-cloud management, failure recovery, and traffic scheduling. Karmada supports deployments on-prem, at the edge, and on public clouds. This guide explains using Karmada with Vultr Kubernetes Engine (VKE). When you follow this guide, you will: * Create three VKE clusters in Vultr's Mumbai, Paris, and New York regions. * Create a fourth VKE cluster for Karmada to manage the rest of the nodes. * Install the Karmada management system on the cluster manager and Karmada agent on other clusters. * Deploy an example application and distribute the workload across all clusters. ## Prerequisites Before beginning this guide, you should: * Have the `kubectl` CLI installed and configured on your local machine. * Have the [Helm client](https://helm.sh/docs/using_helm/#installing-the-helm-client) installed on your local machine. ## Install Karmada API Server 1. [Deploy four VKE clusters at Vultr](https://docs.vultr.com/vultr-kubernetes-engine/#How_to_Deploy_a_VKE_Cluster) with at least three nodes each and download the respective kubeconfig files. * Deploy one cluster in Mumbai. Name the kubeconfig `kubeconfig-mumbai`. * Deploy one cluster in Paris. Name the kubeconfig `kubeconfig-paris`. * Deploy one cluster in New York. Name the kubeconfig `kubeconfig-newyork`. * Deploy one cluster in any other location. Name the kubeconfig `kubeconfig-cluster-manager`. 1. Add the Helm repository to your local machine. # helm repo add karmada-charts https://raw.githubusercontent.com/karmada-io/karmada/master/charts 1. Next, verify the added repository. # helm repo list Sample output: NAME URL karmada-charts https://raw.githubusercontent.com/karmada-io/karmada/master/charts 1. Karmada API server must be reachable from other clusters. So you will require a public IP address of the cluster manager node to expose it for external access. First, retrieve the IP address of the cluster manager. # kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="ExternalIP")].address}' --kubeconfig=kubeconfig-cluster-manager The example output is shown below. Note that `192.0.2.100` is an example IP address for documentation. You'll substitute that address with the actual address of your cluster throughout this guide. 192.0.2.100 1. Install the Karmada control plane on the cluster manager node. # helm install karmada karmada-charts/karmada --kubeconfig=kubeconfig-cluster-manager --create-namespace --namespace karmada-system -- version=1.2.0 --set apiServer.hostNetwork=false --set apiServer.serviceType=NodePort --set apiServer.nodePort=32443 --set certs.auto.hosts[0]="kubernetes.default.svc" --set certs.auto.hosts[1]="*.etcd.karmada-system.svc.cluster.local" --set certs.auto.hosts[2]="*.karmada-system.svc.cluster.local" --set certs.auto.hosts[3]="*.karmada-system.svc" --set certs.auto.hosts[4]="localhost" --set certs.auto.hosts[5]="127.0.0.1" --set certs.auto.hosts[6]="192.0.2.100" > **Note:** Replace the IP `192.0.2.100` with the IP address of cluster manager. Sample output: NAME: karmada LAST DEPLOYED: Sat Nov 19 22:18:54 2022 NAMESPACE: karmada-system STATUS: deployed REVISION: 1 TEST SUITE: None 1. Download the kubeconfig configuration file from the cluster manager to connect to the Karmada API: # kubectl get secret karmada-kubeconfig --kubeconfig=kubeconfig-cluster-manager -n karmada-system -o jsonpath={.data.kubeconfig} | base64 -d > karmada-config 1. Edit the downloaded kubeconfig file: # nano karmada-config Find the following line: server: https://karmada-apiserver.karmada-system.svc.cluster.local:5443 # <- this works only in the cluster Replace that line with the cluster manager node's IP address so that it will work outside the cluster network. server: https://192.0.2.100:32443 ## Install Karmada Agent on Three Clusters In this section, you will install the Karmada agent using Helm on other clusters and join them to the Karmada cluster. 1. First, verify the active status of all clusters. # kubectl get pods -A --kubeconfig=kubeconfig-mumbai # kubectl get pods -A --kubeconfig=kubeconfig-paris # kubectl get pods -A --kubeconfig=kubeconfig-newyork 2. Next, copy the `agent.kubeconfig.caCrt`, `agent.kubeconfig.crt`, and `agent.kubeconfig.key` value from the `karmada-config` file, decode the value using [base64](https://www.base64decode.org/) and store the decoded values in variables named `ca`, `crt` and `key`. 3. Run the following commands to install the Karmada agent on each cluster and link it to the cluster manager. # helm install karmada karmada-charts/karmada --kubeconfig=kubeconfig-mumbai --create-namespace --namespace karmada-system --version=1.2.0 --set installMode=agent --set agent.clusterName=mumbai --set agent.kubeconfig.caCrt="$ca" --set agent.kubeconfig.crt="$crt" --set agent.kubeconfig.key="$key" --set agent.kubeconfig.server=https://192.0.2.100:32443 # helm install karmada karmada-charts/karmada --kubeconfig=kubeconfig-paris --create-namespace --namespace karmada-system --version=1.2.0 --set installMode=agent --set agent.clusterName=paris --set agent.kubeconfig.caCrt="$ca" --set agent.kubeconfig.crt="$crt" --set agent.kubeconfig.key="$key" --set agent.kubeconfig.server=https://192.0.2.100:32443 # helm install karmada karmada-charts/karmada --kubeconfig=kubeconfig-newyork --create-namespace --namespace karmada-system --version=1.2.0 --set installMode=agent --set agent.clusterName=newyork --set agent.kubeconfig.caCrt="$ca" --set agent.kubeconfig.crt="$crt" --set agent.kubeconfig.key="$key" --set agent.kubeconfig.server=https://192.0.2.100:32443 > **Note:** Replace the `--kubeconfig` value with your kubeconfig file of each cluster, `agent.clusterName` value with each cluster name, and `192.0.2.100` with the IP address of the cluster manager. 4. Verify the Karmada agent installation. # kubectl get clusters --kubeconfig=karmada-config Sample output: NAME VERSION MODE READY AGE mumbai v1.25.4 Pull True 95s newyork v1.25.4 Pull True 7s paris v1.25.4 Pull True 31s ## Create Karmada Policy for Orchestrating Multicluster Deployment In this section, you will create a test deployment, define a policy to assign a workload, submit a workload to Karmada then distribute it across all clusters. 1. First, create a test deployment with three replicas to distribute equally across the three clusters. # nano deployment.yaml Add the following configurations. apiVersion: apps/v1 kind: Deployment metadata: name: hello spec: replicas: 3 selector: matchLabels: app: hello template: metadata: labels: app: hello spec: containers: - image: stefanprodan/podinfo name: hello --- apiVersion: v1 kind: Service metadata: name: hello spec: ports: - port: 5000 targetPort: 9898 selector: app: hello 1. Apply the deployment to the Karmada API server. # kubectl apply -f deployment.yaml --kubeconfig=karmada-config Sample output: deployment.apps/hello created service/hello created 1. Run the following command to verify the deployment. # kubectl get deployments --kubeconfig=karmada-config Sample output: NAME READY UP-TO-DATE AVAILABLE AGE hello 0/3 0 0 54s 1. Let's describe the deployment with detailed information. # kubectl describe deployment hello --kubeconfig=karmada-config Sample output: Name: hello Namespace: default CreationTimestamp: Wed, 23 Nov 2022 14:25:56 +0530 Labels: Annotations: Selector: app=hello Replicas: 3 desired | 0 updated | 0 total | 0 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=hello Containers: hello: Image: stefanprodan/podinfo Port: Host Port: Environment: Mounts: Volumes: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ApplyPolicyFailed 106s resource-detector No policy match for resource As you can see, there isn't a policy to match resources. So Karmada doesn't know what to do with the deployments. 1. You will create a policy to assign a replica to each cluster and allocate workloads to clusters. # nano policy.yaml Add the following configuration to define an equal weight for each cluster. apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: hello-propagation spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: hello - apiVersion: v1 kind: Service name: hello placement: clusterAffinity: clusterNames: - mumbai - paris - newyork replicaScheduling: replicaDivisionPreference: Weighted replicaSchedulingType: Divided weightPreference: staticWeightList: - targetCluster: clusterNames: - newyork weight: 1 - targetCluster: clusterNames: - paris weight: 1 - targetCluster: clusterNames: - mumbai weight: 1 1. Apply the policy to the Karmada cluster. # kubectl apply -f policy.yaml --kubeconfig=karmada-config Sample output: propagationpolicy.policy.karmada.io/hello-propagation created 1. Verify whether the Karmada assigned a pod to each cluster using the following commands. # kubectl get deployments --kubeconfig=karmada-config Sample output: NAME READY UP-TO-DATE AVAILABLE AGE hello 3/3 3 3 6m20s To verify the Mumbai cluster, run: # kubectl get pods --kubeconfig=kubeconfig-mumbai Sample output: NAME READY STATUS RESTARTS AGE hello-588ddff5d8-xlw5b 1/1 Running 0 75s To verify the Paris cluster, run: # kubectl get pods --kubeconfig=kubeconfig-paris Sample output: NAME READY STATUS RESTARTS AGE hello-588ddff5d8-qntt5 1/1 Running 0 90s To verify the New York cluster, run: # kubectl get pods --kubeconfig=kubeconfig-newyork Sample output: NAME READY STATUS RESTARTS AGE hello-588ddff5d8-d9pb8 1/1 Running 0 104s As you can see, Karmada assigned an equal number of pods to each cluster. 1. Next, run the following command to scale the deployment to 10 replicas. # kubectl scale deployment/hello --replicas=10 --kubeconfig=karmada-config 1. Verify the deployment. # kubectl get deployments --kubeconfig=karmada-config Sample output: NAME READY UP-TO-DATE AVAILABLE AGE hello 10/10 10 10 12m 1. Verify the age of each cluster. For the Mumbai cluster, run the following: # kubectl get pods --kubeconfig=kubeconfig-mumbai Sample output: NAME READY STATUS RESTARTS AGE hello-588ddff5d8-2xtlb 1/1 Running 0 83s hello-588ddff5d8-65qdx 1/1 Running 0 83s hello-588ddff5d8-p4gdr 1/1 Running 0 83s hello-588ddff5d8-xlw5b 1/1 Running 0 7m32s For the Paris cluster, run the following: # kubectl get pods --kubeconfig=kubeconfig-paris Sample output: NAME READY STATUS RESTARTS AGE hello-588ddff5d8-bc7ct 1/1 Running 0 97s hello-588ddff5d8-qntt5 1/1 Running 0 7m46s hello-588ddff5d8-vkcf8 1/1 Running 0 97s For the New York cluster, run the following: # kubectl get pods --kubeconfig=kubeconfig-newyork Sample output: NAME READY STATUS RESTARTS AGE hello-588ddff5d8-d9pb8 1/1 Running 0 8m1s hello-588ddff5d8-fm7df 1/1 Running 0 112s hello-588ddff5d8-lxk2j 1/1 Running 0 112s 1. Now, edit the `policy.yaml` file so that the Mumbai and Paris clusters hold 40% of the pods and only 20% is left to the New York cluster. # nano policy.yaml Change the configuration as shown below. apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: hello-propagation spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: hello - apiVersion: v1 kind: Service name: hello placement: clusterAffinity: clusterNames: - mumbai - paris - newyork replicaScheduling: replicaDivisionPreference: Weighted replicaSchedulingType: Divided weightPreference: staticWeightList: - targetCluster: clusterNames: - newyork weight: 1 - targetCluster: clusterNames: - paris weight: 2 - targetCluster: clusterNames: - mumbai weight: 2 1. Apply the policy to the Karmada cluster. # kubectl apply -f policy.yaml --kubeconfig=karmada-config 1. Verify the pod distribution again using the following command. For the Mumbai cluster, run the following: # kubectl get pods --kubeconfig=kubeconfig-mumbai Sample output: NAME READY STATUS RESTARTS AGE hello-588ddff5d8-2xtlb 1/1 Running 0 4m14s hello-588ddff5d8-65qdx 1/1 Running 0 4m14s hello-588ddff5d8-p4gdr 1/1 Running 0 4m14s hello-588ddff5d8-xlw5b 1/1 Running 0 10m For the Paris cluster, run the following: # kubectl get pods --kubeconfig=kubeconfig-paris Sample output: NAME READY STATUS RESTARTS AGE hello-588ddff5d8-qntt5 1/1 Running 0 10m hello-588ddff5d8-vkcf8 1/1 Running 0 4m30s For the New York cluster, run the following: # kubectl get pods --kubeconfig=kubeconfig-newyork Sample output: NAME READY STATUS RESTARTS AGE hello-588ddff5d8-d9pb8 1/1 Running 0 10m hello-588ddff5d8-fm7df 1/1 Running 0 4m42s hello-588ddff5d8-lxk2j 1/1 Running 0 4m42s hello-588ddff5d8-vd9lv 1/1 Running 0 89s 1. At this point, pods are running and distributed across three clusters. You can inspect the service in Karmada using the following command. # kubectl describe service hello --kubeconfig=karmada-config Sample output: Name: hello Namespace: default Labels: propagationpolicy.karmada.io/name=hello-propagation propagationpolicy.karmada.io/namespace=default Annotations: Selector: app=hello Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.101.68.149 IPs: 10.101.68.149 Port: 5000/TCP TargetPort: 9898/TCP Endpoints: Session Affinity: None Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ApplyPolicyFailed 16m resource-detector No policy match for resource Normal ApplyPolicySucceed 11m resource-detector Apply policy(default/hello-propagation) succeed Normal SyncSucceed 11m execution-controller Successfully applied resource(default/hello) to cluster mumbai Normal SyncSucceed 11m execution-controller Successfully applied resource(default/hello) to cluster newyork Normal SyncSucceed 11m execution-controller Successfully applied resource(default/hello) to cluster paris Normal SyncWorkSucceed 2m4s (x9 over 11m) binding-controller Sync work of resourceBinding(default/hello-service) successful. Normal AggregateStatusSucceed 2m4s (x9 over 11m) binding-controller Update resourceBinding(default/hello-service) with AggregatedStatus successfully. Normal ScheduleBindingSucceed 2m4s (x13 over 11m) karmada-scheduler Binding has been scheduled ## Conclusion You've finished deploying multi-location, multi-cluster Kubernetes Orchestration with Karmada. You can now use Karmada to automate multi-cluster application management in multi-cloud and hybrid-cloud scenarios. For more information, check out the Karmada official [documentation](https://github.com/karmada-io/karmada/).

Articles in category: Container Orchestration