ESB Performance Round 7.5 - The Other Side of The Story

This blog post explains why the message corruptions stated in “ESB Performance Testing - Round 7” and “Why the Round 6.5 results published by WSO2 is flawed” article are not so catastrophic. Moreover, As you go through the post you’ll understand the fact that, it is written in an absurd manner with overly exaggerated statements. However, with this blog post I don’t really intend to play the same game of theirs but to clear any possible misunderstandings that were caused by those articles. 

Fastest open source ESB in the world

Latest performance study conducted by WSO2 ESB team has clearly showed that WSO2 ESB has continued to be the leader in the space of ESB performance. Geared with latest technology and a dedicated team, WSO2 ESB always provides nothing but the best for its users. Following graph shows the summary of the latest results. For more information please refer Performance round 7.5.

However, There have been some invalid critics on the Net which gives the message that WSO2 ESB fails to deliver. This message is entirely not true and below paragraphs explain why. 

The extinct issue of StreamingXpath 

We must admit that enabling StreamingXpath did lead into a message corruption when the message size is larger than 16K. While there was a real issue here, this was never a default configuration and has NOT really affected the thousands of real deployments of WSO2 ESB out there. Furthermore, this has been stabilised in the recently released WSO2 ESB 4.8.1 as it continues to be the fastest open source ESB.

XSLT and FastXSLT false alarm

XSLT and FastXSLT mediators never had a problem of message corruption. The message corruptions that were seen in Performance round 7 were due to a missing Synapse configuration. Given the fact that, the engineers who conducted the performance test were ex-WSO2 ESB team engineers, they could have easily figured it out and fixed it during the Performance round 7. Plus, they could have informed us about this prior to the test. So that we could have fixed it for them. 

They failed to do neither of these. So, as they have mentioned, their peformance test does have inherent limitations due to their limited understanding. Therefore, it cannot be attributed as a message corruption of WSO2 ESB 4.6 or WSO2 ESB 4.7.0. 

Stability of Passthrough Transport (PTT)

Over the last year WSO2 ESBs with PTT were deployed in many customer sites and they have never encountered any significant issues but rather benefited from high performance of deployed ESBs as the deployment only required very few instances of ESBs. 

To clear any confusions PTT never had message corruption problems but instead StreamingXpath which is written on top of PTT in order to utilize its high performance architecture.

Nothing to Worry

After all, As above section explains message corruptions that were discussed in performance round 7 are either occurs in extreme situations or never really exist. Therefore, we believe the content of the article performance round 7 is more or less misleading the audience. However, StreamingXpath did have a problem with messages larger than 16K which is fixed in ESB 4.8.1. Apart from that there aren’t any message corruptions issues at all.

Lastly, The only other critic that worth answering is why we didn’t publish the AMI. Yes, we didn’t publish the AMI but we did publish the configuration files along with clean and clear instructions to re-setup setup if needed. So, If one wants to reproduce the result they can simply re-setup the setup. Besides, Even if we had published the AMI, one would have to load this AMI into an EC2 instance which is always not guaranteed to be the same. 

As a conclusion, Most of the things that have been published in those articles are trivial stuff and just overly exaggerated to make a big thing out of nothing. However, I must admit some of the critics they have mentioned were really helpful for us to improve our product and I am grateful to them for those.

Read More

Advancing Integration Competency and Excellence with the WSO2 Integration Platform

I am glad that WSO2Con Asia 2014 is held in Sri Lanka. Undoubtedly, It is like the biggest SOA (Service Oriented Architecture) conference that ever held in Sri Lanka. Not only you get to learn anything and everything about SOA but also you get to learn it with hands-on sessions. We all know that best way to learn something is to try it out yourself. So, this is the very best reason why you should attend the tutorial session on "Advancing Integration Competency and Excellence with the WSO2 Integration Platform" done by Dushan and Shammi. 

Mainly, this tutorial session will be focused on the followings,

• New WSO2 ESB Cloud Connectors
• New RESTful Integration capabilities
• Store and Forward and advanced integration patterns

These are some of the latest additions that were done to our ESB. If you find these words unfamiliar, don't worry!. Because you will get to learn from the best. Just to get you started, I'll give a brief introduction on main topics. 

Let's start with WSO2 ESB Cloud Connectors

Here, the million dollar question would be what is a cloud connector ? right. In a sentence "A connector is a ready made and convenient tool to reach publicly available Web API’s". For instance, we have connectors for SalesForce, Google Spreadsheet, Twitter, etc. These connectors allow you to do rapid and easy integration of different APIs to meet business needs. For instance, you can take data from SalesForce and present it Google SpredSheet in minutes. There is no need to write a single code. In fact, it is just a matter of drag and drop from DevStudio. Furthermore, If you don't like these connectors you can write your own connectors. So, in this tutorial you will get use and write connectors. 

New RESTful Integration capabilities

REST is like the next big thing when it comes to integration. Not only it is simple and easy with its "verbs" and "nouns" but also it gives you the liberty of using fat free message types such as JSON, POX, etc as opposed to Web Services. In this tutorial session you will find out that how easy it is to do integration in RESTful manner using WSO2 ESB. To make your life even more easier, the new versions of ESB has enhanced JSON support such as Natural JSON and JSON path. Thefore, this is a tutorial session that shouldn't be missed.

Store and Forward and advanced integration patterns

Though Store and Forward support has been there for some time. We though of revamping its implementation from scratch to cater the modern needs in integration. Store and Forwarding not only helps you to throttle messages but also to archive guaranteed delivery. With this you can do advance EIPs (Enterprise Integration Pattern) such as DLC (Dead Letter Channel) and many more. Moreover, you will get hands-on experience on new features of Store and Forward and its usage in EIPs.

These are the main focuses of this tutorial session. So get involved and you will start to see a set of whole new possibilities in the space of Integration. This could take your organization to the next level. Remember, this is only about a tutorial session. There are series of interesting sessions lined up in WSO2Con Asia 2014. For more information see WSO2Con Asia 2014 official website.

Read More

WSO2 ESB Passthrough Transport Basics

When I first joined WSO2 I found it hard to get a grasp of this so-called "Passthrough Transport". All I knew was it was fast! as opposed to "nhttp transport" (I hadn't known anything about it either). However, over the past year I gradually get to understand what is this Passthrough transport and why it was so fast. So, In this blog post I'll be explaining some "good-to-know" stuff about "Passthrough Transport". Since I am no expert on this, there could be few gaps. But still better than nothing.

Passthrough Transport Vs NHTTP Transport

The main difference is, in Passthrough Transport the incoming message does not get built all the time whereas in NHTTP Transport it always gets built. What we meant by building the message is we take the message stream from the socket and transform it in to a XML representation. 

In reality, you don't always have to build the message. For instance, you maybe able to rout the message simply by looking at the headers of HTTP request. So rather than blindly building the message Passthrough Transport does this selectively which makes it smarter than its predecessor NHTTP Transport.

The main similarity between these two is they are both developed on top of the popular Apache project HTTP-core.

High level view of Passthrough Transport

OK, Now that you have an idea, let's look at high level view this transport.

It is not as simple as the diagram depicts but it is enough to get you started. In a way, Passthrough Transport is a complex implementation of Producer-Consumer pattern. Why I say so, Let me explain. Everything starts from the SourceHandler side. When a client sends a request it comes through HTTP-Core towards SourceHanlder. Then the SourceHandler starts producing data to the Pipe. As soon as SourceHanlder starts producing data to the Pipe TargetHandler starts consuming data from the Pipe. These consumed data are sent through HTTP-Core to the desired endpoint. 

As I said earlier it is not as simple as that. There are quite a lot of classes associated with the process such as ServerWorker, SourceRequest, SourceResponse, ClientWorker, TargetResponse, TargetRequest, etc. To make matters worse, the entire implementation is done in Asynchronous manner.

State Machine of Passthrough Transport

So in order to reduce the complexity of the entire process it is implemented on a state machine. SourceHanlder and TargetHanlder have their separate state machines. Following is the state machine that used by them.

The vertical split represents the SourceHandler side and the TargetHandler side. The horizontal split represents the HTTP request and HTTP response. The methods that are next to each state are the methods that get executed in each state (Forget the methods for the moment). Before I explain the state machine it is important to know that in Passthrough the HTTP message is divided into two parts and as HEADERs and BODY. 

This is how it goes, First ESB establishes the connection with the Client and sets its state to REQUEST_READY. Then it starts receiving data. First it reads the headers and goes to REQUEST_HEAD state. Afterwards, it makes itself ready to read the body of the message in the REQUEST_BODY state. Finally to finish the first quarter it reads the entire message body and moves to REQUEST_DONE state. 

The same thing continues in the next quarter but this time ESB acts as the client to some back-end server.  And in the last two quarters it happens for the response. The states of the SourceHandler and the TargetHanlder are interconnected through Pipe's buffer. So sometimes when you debug, though the error message shows in TargetHandler side the actual cause could be on the other side. 

Exact Location of Passthrough Transport

Following diagram shows the location of the Passthrough Transport in the ESB architecture.

Finally, THIS STATE MACHINE REPRESENTS AN IDEAL SCENARIO. But in real world it could deviate a bit from this. Moreover, this blog post is only covering a small bite of a complex implementation. Anyways, this knowledge is enough to get you started. 

Read More

Exchanging SAML2 token to OAuth2 token in WSO2 Platform

This is something I came-across while I was on my first QSP. There can be situations where you need to exchange SAML2 to OAuth2 token. In our case, we authenticate users using SAML2 and then authorize APIs on behalf of the user using OAuth2. In this blog post, I'll be walking through how this type scenario can be handled using WSO2 products [1]. In fact, I'll be using WSO2 Identity Server, WSO2 ESB and WSO2 API Manger. Firstly, lets start with the deployment diagram. So, that everyone can get a grasp on how these components are connected to each other and the order of the communication.

As depicted in the image. User get authenticated with SAML2 token and then Service Provider exchanges it to OAuth2 token using the API manager. Now let's see how to configure these components to archive this [2].

STEP 1 - Configure Identity Server (IS 4.6.0)

Register the Service Provider as in the image for authenticating using SAML2. For more information refer link [3]. For Assertion Consumer URL enter http://localhost:8080/travelocity.com/samlsso-home.jsp and be cautious not to forget to select wso2carbon for Certificate Alias. 

STEP 2 - Configure API Manager (APIM 1.6.0)

In this example, we will be configuring all the products in one machine. Therefore, Let's in crease port offset by 2. In order to do that open <APIM_HOME>/repository/conf/carbon.xml and set the port set as below. Now start the APIM.

<!-- Ports offset. This entry will set the value of the ports defined below 
to the define value + Offset.  e.g. Offset=2 and HTTPS port=9443 will
set the effective HTTPS port to 9445 -->
<Offset>2</Offset>

After that in APIM go to configur and click on Trusted Identity Providers. There fill the fields as in the below image.

For the Identity Provider Public Certificate there are two important things to do. First, we need to generate a certificate and then we need to add that certificate to JAVA trusted certificates. So, please issue following commands accordingly. 

• keytool -export -alias wso2carbon -keystore <IS_HOME>/repository/resources/security/wso2carbon.jks -storepass wso2carbon -file mycert.pem
• keytool -import -trustcacerts -file <IS_HOME>/repository/resources/security/mycert.pem -alias wso2carbon -keystore $JAVA_HOME/jre/lib/security/cacerts

STEP 3 - Modifying Service Provider  (travelocity.com)

In travelocity.com when you fist get authenticated with SAML2 you get the SAML2 token. Therefore, Using that SAML2 token you are going to get the OAuth2 token. So, that is what done in the below code snippet.

Exchanging SAML2 token to OAuth2 token

 // Get the SAML2 Assertion part from the response
StringWriter rspWrt = new StringWriter();
XMLHelper.writeNode(samlResponse.getAssertions().get(0).getDOM(), rspWrt);
String requestMessage = rspWrt.toString();

// Get the Base64 encoded string of the message
// Then Get it prepared to send it over HTTP protocol
String encodedRequestMessage = Base64.encodeBytes(requestMessage.getBytes(), Base64.DONT_BREAK_LINES);
String saml2assertion = URLEncoder.encode(encodedRequestMessage,"UTF-8").trim();

String urlParameters = "grant_type=urn:ietf:params:oauth:grant-type:saml2-bearer&assertion=" + saml2assertion + "&scope=PRODUCTION";

//Create connection to the Token endpoint of API manger
url = new URL("https://localhost:9445/oauth2/token");

connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8");
// Set the consumer-key and Consumer-secret
connection.setRequestProperty ("Authorization", "Basic " + Base64.encodeBytes(("0P6YbqXQHwS38rTJ5wIzzrIUgNga:HosDgUAhLrgoZh2Ts_L2nrzf4V0a").getBytes(), Base64.DONT_BREAK_LINES));
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);

//Send request
DataOutputStream wr = new DataOutputStream (connection.getOutputStream());
wr.writeBytes (urlParameters);
wr.flush ();
wr.close ();

//Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));

String line;
StringBuffer response = new StringBuffer();
while((line = rd.readLine()) != null) {
   response.append(line);
   response.append('\r');
}

rd.close();
return response.toString();

As you may have already noticed you need a consumer key and a consumer secret in order to get the OAuth2 token. So this consumer key and consumer secret are retrieved when you get subscribed to a particular API available in the APIM store.

Following is an sample SAML2 Assertion which was taken from SAML2 token. As mentioned in the code snippet, you only need this part to get OAuthe2 token.

<saml2:Assertion xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" ID="lkombgamkgmffhiaphjlipbgdmlnigdgbgmhidpi" IssueInstant="2014-01-16T15:20:09.230Z" Version="2.0">
   <saml2:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">https://localhost:9443/samlsso</saml2:Issuer>
   <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
      <ds:SignedInfo>
         <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
         <ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" />
         <ds:Reference URI="#lkombgamkgmffhiaphjlipbgdmlnigdgbgmhidpi">
            <ds:Transforms>
               <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
               <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
            </ds:Transforms>
            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
            <ds:DigestValue>CaY1tbi2kfzCqnJARZBs9I6C690=</ds:DigestValue>
         </ds:Reference>
      </ds:SignedInfo>
      <ds:SignatureValue>dOExwKi/lAW7nzb2JCyLJCAppI9sgb0qZDayQcNeiSqv3gjRmsOcfxYyeVZhUaqHuOpqCqWwLQDQ
i4BUINMdlBsw8y2iZH7bhcfUgDIj26PNBlFtZthmX3ERr4leCm0NIo0jt+cVry3BSEO7duamNq3J
ZPIultt6SZWTsfk4nn8=</ds:SignatureValue>
      <ds:KeyInfo>
         <ds:X509Data>
            <ds:X509Certificate>MIICNTCCAZ6gAwIBAgIE...<removed for bravity>...O4d1DeGHT/YnIjs9JogRKv4XHECwLtIVdAbIdWHEtVZJyMSktcyysFcvuhPQK8Qc/E/Wq8uHSCo=</ds:X509Certificate>
         </ds:X509Data>
      </ds:KeyInfo>
   </ds:Signature>
   <saml2:Subject>
      <saml2:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress">admin</saml2:NameID>
      <saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
         <saml2:SubjectConfirmationData InResponseTo="0" NotOnOrAfter="2014-01-16T15:25:09.230Z" Recipient="http://localhost:8080/travelocity.com/samlsso-home.jsp" />
      </saml2:SubjectConfirmation>
   </saml2:Subject>
   <saml2:Conditions NotBefore="2014-01-16T15:20:09.230Z" NotOnOrAfter="2014-01-16T15:25:09.230Z">
      <saml2:AudienceRestriction>
         <saml2:Audience>travelocity.com</saml2:Audience>
         <saml2:Audience>https://localhost:9445/oauth2/token</saml2:Audience>
      </saml2:AudienceRestriction>
   </saml2:Conditions>
   <saml2:AuthnStatement AuthnInstant="2014-01-16T15:20:09.230Z" SessionIndex="28350436-898a-42c4-975f-e8b5aba01d9a">
      <saml2:AuthnContext>
         <saml2:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:Password</saml2:AuthnContextClassRef>
      </saml2:AuthnContext>
   </saml2:AuthnStatement>
</saml2:Assertion>

Making the Service Call

Now you have everything that you need for the legitimate service call. All you have to do is use the retrieved OAuth token to make service call. Following code snippet shows how it is done.

// Create the connection to desired API
url = new URL("http://localhost:8282/datadelete/1.0.0");
            
connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
// Using the OAuth token
connection.setRequestProperty ("Authorization", "Bearer " + request.getSession().getAttribute("access_token"));
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);

//Send request with the required payload
DataOutputStream wr = new DataOutputStream (connection.getOutputStream());
wr.writeBytes ("{\"Request\":{\"DeviceID\":\""+ request.getParameter("device") +"\"}}");
wr.flush ();
wr.close ();

//Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));

String line;
while((line = rd.readLine()) != null) {
    rsp.append(line);
    rsp.append('\r');
}
rd.close();

return rsp.toString();

So that is it. That's how you can exchange SAML2 token to OAuth2 token in WSO2 platform. Anyways, you might be wondering why there is an ESB. The actually backend service is hosted in ESB and then exposes using API manager.

You can also exchange the SAML2 token to OAuth2 token just using the IS instead of APIM [4].

See Also

[1] http://docs.wso2.org/dashboard.action
[2] http://docs.wso2.org/display/AM160/Token+API
[3] http://docs.wso2.org/display/IS460/Configuring+SAML2+SSO
[4] http://docs.wso2.org/display/IS450/SAML2+Bearer+Assertion+Profile+for+OAuth+2.0

Read More