Getting Started: Cluster Deployment

This guide walks you through deploying a VirtualMetric cluster, which provides high availability, scalability, and load balancing across multiple nodes.

Prerequisites

Before starting a cluster deployment, ensure you have:

• Multiple servers or VMs for node deployment
• Network connectivity between all nodes
• Sufficient resources on each node (CPU, memory, storage)
• Required ports open in firewalls between nodes (typically 4222 for client connections and 6222 for cluster communications)

Cluster Architecture

A VirtualMetric cluster consists of:

• Environment: Logical grouping of resources (e.g. datacenter, region)
• Cluster: Group of nodes that work together
• Nodes: Individual servers running the VirtualMetric service

Configuration Structure

Create a cluster.yml file that defines your cluster environment. The example below shows a two-node cluster configuration in the "NewYork" environment:

environments:  
  - name: "NewYork"
    description: "Datacenter1"
    status: true
    clusters:
      - name: "CLS01"
        description: "Datacenter1 CLS01"
        status: true
        type: director
        properties:
          address: "0.0.0.0"
          port: 6222
          authentication:
            username: "user1"
            password: "password1"
          tls:
            cert_name: ""
            key_name: ""
            ca_name: ""
            verify: true
            status: false
        nodes:
          - name: "Node01"
            description: "10.0.0.1 ip address"
            status: true
            properties:
              address: "0.0.0.0"
              port: 4222
              cluster:
                address: "0.0.0.0"
                port: 6222
          - name: "Node02"
            description: "10.0.0.2 ip address"
            status: true
            properties:
              address: "0.0.0.0"
              port: 4223
              cluster:
                address: "0.0.0.0"
                port: 6223
                authentication:
                  username: "user2"
                  password: "password2"

Configuration Parameters

Environment Level

Parameter	Description
name	Unique name for the environment (e.g. datacenter or region)
description	Human-readable description
status	Enable/disable the environment (true/false)

Cluster Level

Parameter	Description
name	Unique identifier for the cluster
description	Human-readable description
status	Enable/disable the cluster (true/false)
type	Cluster role (e.g. "director" for primary cluster)
properties.address	Bind address for cluster communications
properties.port	Port for cluster communications (default: 6222)
properties.authentication	Cluster-level authentication credentials
properties.tls	TLS configuration for secure cluster communications

Node Level

Parameter	Description
name	Unique identifier for the node
description	Human-readable description (often includes IP or hostname)
status	Enable/disable the node (true/false)
properties.address	Bind address for client connections
properties.port	Port for client connections (default: 4222)
properties.cluster	Node-specific cluster configuration

Deployment Steps

1. Prepare Configuration

1. Create the cluster configuration file as shown above
2. Customize environment, cluster, and node names
3. Configure appropriate IP addresses and ports
4. Set authentication credentials if needed

2. Deploy to Each Node

For each node in your cluster:

1. Install the VirtualMetric service package
2. Copy the same configuration file to each node
3. Start the service with the appropriate node name

For example, on Node01:

./virtualmetric -path /path/to/cluster.yml -node Node01 -cluster CLS01 -region NewYork

And on Node02:

./virtualmetric -path /path/to/cluster.yml -node Node02 -cluster CLS01 -region NewYork

The service will:

• Read the configuration file
• Identify its node configuration based on the provided node name
• Start the appropriate services
• Connect to other nodes in the cluster

3. Verify Cluster Formation

After starting all nodes, the cluster will begin to form. The service automatically:

1. Establishes connections between nodes
2. Waits for cluster functionality (visible in logs)
3. Monitors cluster health continuously
4. Attempts to recover from failures

You can verify cluster formation by checking logs for messages indicating successful cluster connections:

VMMQ cluster service is active. All peers connected.

Security Considerations

Authentication

It's recommended to enable authentication for production clusters:

authentication:
  username: "cluster_user"
  password: "secure_password"

Different authentication credentials can be set at both cluster and node levels.

TLS Encryption

Enable TLS to secure communications:

tls:
  status: true
  cert_name: "server.crt"
  key_name: "server.key"
  ca_name: "ca.crt"
  verify: true

Place certificate files in the service root directory on each node.

High Availability Features

The VirtualMetric cluster includes several built-in HA features:

Automatic Failure Detection

• Monitors node health every second
• Updates health status when problems are detected
• Logs warnings for cluster issues

Self-Healing

• Automatically attempts to restart failed server components
• Monitors for shutdown signals and performs graceful shutdown
• Waits for cluster functionality on startup

Monitoring

Three monitoring routines run continuously:

1. Server Monitor: Checks server health and restarts if needed
2. Cluster Monitor: Verifies cluster functionality
3. Shutdown Monitor: Ensures graceful shutdown when requested

Troubleshooting

Common Issues

1. "Cannot find node information" error:

   • Ensure the node name provided with the -node parameter exactly matches a node name in your configuration

2. Nodes can't connect:

   • Verify network connectivity between nodes
   • Check firewall settings for cluster ports
   • Ensure cluster addresses and ports are correctly configured

3. Cluster warning messages:

   • "VMMQ cluster service is not active. Waiting for other peers..."
     • This is normal during startup or when some nodes are offline
     • Check that all nodes are running and can connect to each other

4. Authentication failures:

   • Verify credentials match in all node configurations
   • Check for typos in usernames or passwords

Scaling the Cluster

To add more nodes:

1. Add new node definitions to the configuration file
2. Deploy the updated configuration to all nodes
3. Start the service on the new nodes

The cluster will automatically incorporate the new nodes.

Next Steps

After successfully deploying your cluster, consider:

1. Setting up collectors to ingest data across the cluster
2. Configuring data processing pipelines
3. Implementing monitoring for the cluster nodes
4. Setting up backups and disaster recovery procedures

For more advanced configurations and features, refer to the complete documentation.