With the introduction of Cloud, SaaS, and the Internet of Things, and an increase in Outsourcing, Mergers, and Acquisitions; the need for Integration has grown. In today’s highly variable and competitive environment in which change accelerates at a faster pace each year, organisations need to have a holistic set of capabilities in order to compete. As such, Integration is no longer an afterthought for organisations and they are now beginning to include budgets for Integration activities. There is an abundance of available integration technologies from over a hundred vendors that can help, but the important question that decision makers must now ask is “Which technology offers the right solution for us?”
At Mitra Innovation we love to share our knowledge and experience. We are a Preferred Partner of WSO2 and carry out many integration projects using WSO2’s range of modern integration technology products to implement full-fledged enterprise integration solutions according to industry leading standards. These products include
● Advanced full-text search capabilities
● WSO2 API Manager/Gateway
● WSO2 Enterprise Service Bus (ESB)
● WSO2 Message Broker (MB)
● WSO2 Data Analytics Server (DAS)
This article acts as a reference document and supplementary guide for senior engineers and architects, wishing to evaluate the WSO2 application stack evaluation for enterprise grade integration solutions. In addition, it illustrates how the WSO2 analytics platform can be integrated and utilised in a typical integration platform and used as a Business Intelligence tool for administrators and management. Furthermore, this article explains the advantages and capabilities of using WSO2 as an integration middleware platform. Advantages include security, guaranteed information delivery and data analytics capabilities.
Typical Business Requirements for an Integration Solution
In this article we discuss the typical business requirements for the best practices and architectural patterns to be applied in a middleware platform implementation. For the purpose of this article, we highlight the integration and use of a WSO2 analytics platform as an example.
When it comes to integrating two or more services, it can be done either within the corporate LAN or completely outside of it while calling on a service within the corporate LAN. The proposed WSO2 Integration solution in this article is designed to cater to both requirements in the same implementation.
Integration Solution designs are generally divided into two sections which are grouped based on whether the services they provide can be accessed from the public unsecured environment (internet) or not. These sections are:
1.Internal integration solution (LAN bound)
2.External facing integration solution (Demilitarised Zone/DMZ bound)
Internal Integration Solution (LAN bound)
The Internal Integration Solution (LAN bound) is where the primary integration solution should be implemented and is responsible for passing all information back and forth between different third-party software systems that are part of the integration requirement. Integration services implemented in this layer are completely inaccessible from the public internet or any other corporate LAN environment and are fully protected from outside attacks. All the WSO2 products implemented under this group will expose the integration services and will implement the integration business logic.
External Facing Integration Solution (De-Militarised Zone/DMZ bound)
The External Facing Integration Solution layer simply obtains service calls from the unsecured public internet and passes the service calls to the internal integration layer for processing. Services hosted in this layer must be highly secured with different security mechanisms and implementations such as:
1.Trusted CA security certificate configuration with a very strong encryption level
2.Authentication mechanism implementation
3.Whitelisting the servers that are allowed to access the DMZ services
4.Impose throttle limits on service calls appropriately
5.Collect statistics on suspicious activities
6.Restrictions on access to server ports
7.User role and authorisation management from corporate AD servers.
Service calls received to this layer are passed to internal integration services through the corporate firewall and as such, even if the servers in this layer are compromised, the internal integration system is still safeguarded from outside attacks.
When designing an enterprise grade integration solution, it is important to fulfill several non-functional requirements. This is because the platform has to be efficient, reliable and resilient for the integration services it is intended to provide. It is also the case that in an enterprise environment, every system does a specific job and is solely concerned about the integrity and the availability of the information they hold. These non-functional requirements include:
1.Cluster the servers (i.e.WSO2 ESB, API Manager, MB and DAS)to facilitate high availability of the platform services.
2.Use a load balancer to switch traffic to an active node to prevent users from getting timed-out or receiving failure messages due to clustering.
3.Use a resilient DB implementation with minimum RAID level 1.
4.Allow only absolutely necessary ports from the corporate firewall.
5.Use servers with sufficient performance and resources (e.g. CPU/Memory/Disk performance etc.).
6.Use server backups to overcome disasters.
7.Facilitate sufficient network bandwidth and QOS.
The Internal Layer
The internal layer integrates any number of internal services provided by the third-party systems within the same corporate environment (LAN) and passes messages from one to another or to many services (service orchestration) according to the business requirement.
Integration services implemented in the ESB are exposed via API gateways and are available (accessible) only within the corporate LAN environment. Thus, the ESB cluster can act as both a service provider and a service consumer. Corporate third-party services that consume ESB services are upstream services. Whereas the third-party services which the ESB invokes are downstream service providers which the ESB consumes from. Typically any corporate third-party service that consumes integration services accesses API gateway services. These get routed back to the ESB for processing.
The Role of ESB
An integration engine is needed to pass messages between third-party services. This is exactly what WSO2 ESB is designed for. It can communicate with different services using various methods and protocols such as SOAP/Rest, JMS, DB, and Emails or even using flat files. It has the capability to transform and format messages between foresaid types before passing on information to the other service provider (ETL functions).
The Role of API Gateway
All the services that the ESB provides should not be exposed even within the corporate environment. They have to be controlled such that only those people absolutely required to access the ESB integration services are able to do so. In addition, access security and access throttling must be implemented. This is what API gateways are designed for. The WSO2 API gateway which is distributed with product WSO2 API Manager is ideal for this. In addition API gateways are capable of publishing service statistics to the analytics and monitoring tools.
The Role of API Key Manager
Security in the API gateway can be OAuth or Basic Authentication etc. These authentication services are provided by WSO2 Key Manager. Integration services need to be accessed with an OAuth authentication token (provided by the Key Manager in a prior step to access the services) from the API gateway which is validated at the API Key Manager. The Key Manager is responsible for validating this key and authorising or declining the service call. The Key Manager is also responsible for validating the domain users registered in the corporate LDAP server before authorising service calls. This requires the LDAP server to be configured in the API Key Managers as a secondary user store.
The Role of API Manager
API gateways do not have an interface to create APIs or to publish them. The API Manager contains the user interfaces to create/manage/publish and subscribe to the integration service APIs which will be hosted from API Gateway Managers. The WSO2 API Manager product is bundled with API Manager, API gateway and API Key Manager components by default. Thus in this case, the requirement is to leverage a distributed implementation of the module stack which will work independently on their designated role.
The Role of Message Broker
In some instances (if not in most cases), it is required to convey service calls in a reliable manner. As an example, the service consumer may require that the message encapsulated in the service call must reach the intended third-party service provider, but it is not guaranteed that the third-party service provider can handle thousands of calls at once or will be available at a given point of time. This is where the message broker comes into the picture. It sits in the middle, holding the service messages which are delivered in a first-in-first-out (FIFO) order whilst re-trying to deliver messages in case of a failure. A service message will not be lost until it is delivered to the respective third-party service provider. Since the message broker is clustered, messages are still preserved safely in case of a failure in the message broker server itself.
The Role of the Data Analytics Server
All the message flow logs and errors must be collected and should be accessed from a single point in a complex distributed system like this one we are designing. As such, the client or the management may require information analytics of the integration platform. Thus, a data collection and analytics platform which will provide services to feed information, perform analytics on them and generate analytical views and alerts based on predefined requirements, becomes necessary. WSO2 DAS does this job in this scenario.
The Role of the Load Balancers
As mentioned in the main section of this document, all the servers mentioned above need to be clustered for high availability of the platform, and load balancers come into play to manage the service calls in a clustered environment. There will be a load balancer sitting in front of the WSO2 ESB, API Gateway and API Key Manager product clusters to direct the data traffic to one of the available servers or to the server with least load (depending on the configuration). You can optionally choose to implement the WSO2 MB and DAS behind a load balancer. Load Balancers can be Layer-4 or Layer-7 depending on the complexity of the design and the requirement, but it can be Layer-4 at this case. The popular NGinx load balancer is a good option which is free to users.
The Role of the Firewall
The firewall will limit both inbound and outbound port access to facilitate the absolute requirement in terms of the operating system and application ports to allow data transmission. As an example WSO2 ESB uses port 8243 to host services. Therefore this port must be opened for inbound traffic to enable the DMZ layer to send messages into the corporate LAN. It should never be the case to allow all ports or unnecessary or unsure port as this can lead to a serious security issue.
The DMZ layer consists of an API gateway cluster to act as an integration service provider to the outside world (yet restricted as explained in earlier section). It sits behind a separate load balancer. Usually, there should be a public DNS name which routes the traffic to the load balancer where the load balancer re-routes the traffic back to the API gateways based on the load balancer configuration.
It is important to note that all the API services exposed via the DMZ layer API gateways are required to configure with some kind of security authentication mechanism (OAuth or Basic Authentication etc.), with a valid SSL certificate (with a strong encryption level) issued by a trusted certificate authority.
Any service call to the DMZ API Gateway will be calling the web services hosted by the ESB via the firewall which will do the processing of the messages passed into it. In addition, the external gateway cluster will talk to the internal Key Manager cluster for API key validation and also will talk to the LDAP server (Active Directory) for user authentication if required. All these communications are to be allowed to pass through the firewall only with specific configurations that allow only those ports that need to be contacted.
Below is the design of the enterprise integration solution discussed above which may fit most integration requirements. It has complexities ranging from average to advanced with a few tweaks, but can also be extended in any way to suit the actual/specific business requirement.