Hazelcast clustering with WSO2 carbon servers in 20 minutes - part 2

Since first part of this blog post explains about clustering concepts, I would rather directly start this blog post with configuring the servers. The primary focus of this blog post is to show the interaction of well-known-members and rest of the other members. This knowledge is a must to have if you are working with any clustered deployment. Any behavior of this cluster is more or less influenced by the implementation of Hazelcast. Therefore, knowing Hazelcast would always provide you an extra support.

Deployment diagram 

Let's start with the deployment diagram. For this deployment I will be using four ESB instances and two of which would be well-known-members where as the other two are dynamic members. Yes, this is not a real world production deployment but an ideal deployment to understand any production deployment. 

Wondering why two WKAs ?

All right, now you must be wondering why there are two well-known-members and two ordinary members. For a given cluster, it is best  that if we can make all the members as well-known-members. You'll get to know why later. However, this is not practical in reality and as a result we will have to have both dynamic and static members (well-known-member). 

Therefore, we have to elect few members as well-known-members and for this cluster I have elected two well-known-members. This is mainly to avoid single point of failure. Without well-known-members there is no way a new node to join a cluster. So in this case, if one well-known-member goes down still we can keep our cluster pretty much alive as we have another. 

As a rule of thumb, it is always better to have as many WKAs as possible. So, you have the luxury to point dynamic members to well-known-members as many as possible. The higher the well-known-members, the higher the availability.

Configuring the servers

Configuring the well-known-members

Let's start configuring the well-known-members. The only file you have to touch in order to do this is the axis2.xml. Yes, that is the only file. Following is the configuration snippet of the well-known-member 1. 

   
   true
   
   
   wka
   
   wso2.esb.domain
   
   45564
   100
   60

   
   127.0.0.1

   
   4100

   
      
      
   

   
   
      
         127.0.0.1
         4200
      
   

   
      
   
   

I have removed all the default comments and added some new comments to guide you guys through the configurations. Now, we can start well-known-member 1. Since, I have used the same machine for the complete deployment, to start well-known-member 2 all I have to do is changing the localMemberPort port to some unique value. I have changed it to 4200 as follows.

4200

Needless to say when you start multiple carbon servers in the same machine you have to set the port offset. For this deployment, I opted to start the sever with sh  ./bin/wso2server.sh -DportOffset=1. However, there is something you need to know. Changing the port offset does NOT impact localMemberPort. 

Upon successful start you should be able to see something similar to the following in the console.

[2014-08-10 21:55:33,921]  INFO - WKABasedMembershipScheme Member joined [3e823e7e-e897-4625-abb9-7b6c3cca8d1f]: /127.0.0.1:4100
[2014-08-10 21:55:35,963]  INFO - MemberUtils Added member: Host:127.0.0.1, Remote Host:null, Port: 4100, HTTP:8280, HTTPS:8243, Domain: wso2.esb.domain, Sub-domain:__$default, Active:true

Configuring the Dynamic-members

We just have to follow the same step changing the localMemberPort to configure the dynamic members. But there is one additional step to do. In the members list we have to mention about the WKAs. Doing this automatically makes the other two members as WKAs. There is no other special configuration to make a member a WKA member. Following code snippet shows how to do this. 

   
      127.0.0.1
      4100
   
   
      127.0.0.1
      4200
   

All right that is about it. You have your own Hazelcast cluster with WSO2 carbon servers. Now you can enhance the cluster by adding the following,

• A external GREG to share common resources between all the members in the cluster.
• A deployment synchronizer to synchronize artifacts among other members.
• A load balancer to rout the load to each member of this cluster. 

To learn more on WSO2 product clustering see [1].

[1] https://docs.wso2.com/display/CLUSTER420/WSO2+Clustering+and+Deployment+Guide

Read More

Hazelcast clustering with WSO2 carbon servers in 20 minutes - part 1

Introduction 

While we were at customer site, we were bombarded with above subject. We almost chocked ourselves answering those questions. So, I though of writing a blog post based on the experience  and the knowledge I gained during my on-site engagement. This blog post explains everything you need to know about Hazelcast clustering in a production deployment. 

Why do we need clustering

In a typical enterprise deployment we don't deploy a single instance of a given server as it could result in a single point of failure i.e if the deployed server goes down the complete system will be unusable. Thus we always tend to deploy multiple instances of a given server in order to increase the Availability of a system. 

However, this is only one aspect of this. The other aspect of this is Scalability of a given deployment. In modern enterprise systems a single server instance is not enough to cater the number of incoming request. Therefore, in order to scale we always add more instances to the existing system. This is called Horizontal scaling. Though we could also upgrade server specs such as increasing the memory and CPU speed in order to scale (which we call vertical scaling), there is always a limit and whether we like it or not we have to add more instances to scale.

So it is obvious that we need to have multiple instances of a given server in an enterprise deployment. Needless to say adding more instances adds more complexity to the system. In order to be consistent regardless of numbers servers you've added, you may have to replicate the state and make servers communicate with each other and that is where clustering comes to picture.

Clustering Concepts

Membership discovery phase

When you add a new node to an existing system, it has to convert itself to a member of the existing cluster. A member of a cluster knows about each other in the cluster, which allows that member to change its state to match with other core-existing members. There are two mechanisms to become a member of a cluster. A node can either use Well Known Address (WKA) mechanism or Multicast mechanism. Now what are these ?

Multicast mechanism 

In Multicast a node advertises its details to others using a multicast channel. All the other members get to know about the new node through this multicast channel, which allows them to start communicating with the new node. This allows the node to a become member in the cluster. However, Multicast is not preferred for production deployments as it could add an unnecessary overhead to the network. As a result, it is more often use for testing purposes. 

Well Known Address (WKA) mechanism

In WKA there is a set o well known members and everybody knows about these members. When a node wants to become a member of the cluster, it connects to one of the well known members and declare its details. Then the well known member provides all the information about the cluster and let every member in the cluster know about the new node. This allows the node to become a member of the cluster. This is the widely used membership discovery mechanism in clustering.

Static vs Dynamic membership 

A cluster deployment could have static, dynamic or hybrid members. In a static clustered setup there is a fix set of members and it is not possible to add a new member to the cluster without restarting the system. IP address and port number of static members are predefined.  In a Dynamic clustered setup we can always add new members to the system without restarting. However, in Hazelcast we always use a hybrid clustered setup where we have both static and dynamic set of members. Static members are the well known members who have a predefined IP and port. 

Member's view

Each member in the cluster has its own view of the cluster. Once it discovers the members of a cluster it keeps track of these members. Normally, this is done by maintaining a heart-beat pulse between other members. This way when a member goes down it could detect it and remove that member from the healthy list. However, this is also called unreliable failure detection as members may not respond to the heart-beat request due to the load on that member and not because it is really down. 

Clustering domains

This may not come under general clustering concepts but rather specific to WSO2. In order to identify a cluster we label it with a domain name. Clustering messages will only be sent to the members of that particular domain. In addition to that, this way we can route the traffic only to the relevant set of instances.  For example, let's say there is a load balancer fronted with multiple cluster domains of ESB and BPS. Load balancer will look into the domain mapping and route the message to the specific cluster domain. Therefore, ESB requests are isolated from BPS requests and vise versa. 

Now that you have a basic idea about the concepts of clustering, in part 2 I'll be discussing how to configure WSO2 carbon servers using Hazelcast. 

Read More