Fabric Deployment to Profile Git - Automation

Fabric maven plugin offers a wide variety of deployment options for Jboss Fuse. This article explores the deployment method pertaining to modifying the profiles on the Fuse profile GIT.

The two commands which we use to deploy the code in this article are fabric8:zip and fabric8: branch. Documentation on the plugin’s can be found here .

Scenario:

An enterprise project has the below bundles divided in to two profiles.

Common       :  DBCP bundle and MQSql Connector

Web Module : Has Business Related bundles, for e.g., Let us say

·      mvn:com.sundar.fabric8/webbackend/0.0.1-SNAPSHOT

·      mvn:com.sundar.fabric8/webfrontend/0.0.1-SNAPSHOT

·      mvn:com.sundar.fabric8/soapservices/0.0.1-SNAPSHOT

The goal is to generate a profile using fabric8:zip such that all the bundles from common are written to a profile common and the three bundles under the web module are written to web module profile.

Creating Profile Zips

1.   In addition to your regular code maven projects create a new project fabric8-deployment as a parent project.
2.  Create two modules common and web module with any packaging mode you wish (except pom), the main goal here is to generate profile zip files.
3. Modify the parent pom as below.
4. 
   
5. The distribution management section of the pom is needed. I have used a local repository for the example sake , it can be modified accordingly to the needs of the project.
6. In traditional fabric8:zip usage , the fabric8:zip plugin will be associated in the actual bundle builds and thus create a one-one to mapping for a profile and a bundle.
7. In this example the project which defines the fabric8:zip goal is marked to be ignore using the <includeArtifact> tag of the fabric8:zip goal which will then not write the project in to the profile created.
8. Now edit the common module pom to reflect as below.
9. Profile tag in the properties will be used to create the name of the profile.
10. ParentProfiles are mentioned if there is any associated parents for the profiles needed. (common examples are CXF projects might need to be associated with feature-cxf profile).
11. Bundles tag is where all the maven co-ordinates for the bundles and packages which go in to this project are mentioned separated by a space. The common profile in our example needed to have DBCP And MYSQL connectors, so these are added under the bundles tag.
12. Features tag is a place-holder in case any feature need to be added to the tag separated by spaces.
13. Edit the web module pom to reflect as below
14. Now run the fabric8:zip command on the parent project , you should see that the project zips are created individually and aggregated in to the parent target directory
15. You can run the mvn install command to automatically zip the profiles and deploy them to the locations specified in the distributionManagement part of the pom.

The generated profile will look as below

Creating the Project to Deploy the profile to the Git

     Once the profiles are created and installed to a accessible repository, we can now create the fabric8:branch  plugin based project to push these profiles to the repository.

1. Create a project camel-fabric-deployment and edit the pom as below
2. The distribution management section needs to point to the same repository where the profile zip files have been installed for the dependencies to be picked up properly.
3. Add all the profiles, created in the earlier step as dependencies to the project. Observe the classifier and type fields provided to specify the profiles.
4. Provide the branch name as the latest version .
5. Provide the oldBranchName as the current version in the git.
6. PushOnSuccess will determine if the repository needs to be updated on the successful compile or if it has not to be pushed.

Adding any Configuration files to the Profile.

           
We can add the configuration files to the profile within this framework by adding the profileConfigDir to the fabric8:zip goal configurations. The default value for this property is src/main/fabric8 , where in all the configuration files stored in this directory will be copied in to the zip file and subsequently pushed on to the profile via fabric8:branch goal.

            We can even have environment wise files created in each directory and over-ride the value of the profileConfigDir using the system variables. E.g., src/main/fabric8/${environment} and then run the command

Mvn –Denvironment=dev install

To generate the profile zip with only the files under the directory dev to move to the profile. This gives us a cleaner way of managing the property values from environment to environment.

Conclusion: Using the configuration properties provided by the fabric8 maven plugin we can tweak this deployment approach as needed by our environment. This approach has a pit-fall that the json file which describes the dependencies of each project (bundle), which generally gets created by the fabric8: zip goal will not be created. The ignoreArtifact property set to true will over-ride this functionality. Generally depending upon the setup of your project, this way can be safely adopted in to projects running fuse on ensemble in a common set-up.

For Sample Code Click Here

References: Thanks to Richard Stroop ,for clarifying my queries while working on the project.

Using AutoScaler in Fuse

Autoscaling is an easy repeatable way of spinning up fabric containers associated with the profiles . It can be particularly helpful while performing load tests and also to create the containers when new profiles are being deployed to the environment.

• Add the auto-scale profile to the root-container of the fabric (parent container ) not to the child containers.

Profile Requirements  can be defined  in two ways;from the web-console or the command line.

• Command Line : 
• fabric:require-profile-set command is used for defining the requirements for the instances that the profile is applied on 
• fabric:require-profile-set --minimum 1 --maximum 3 some-profile-id which then tells fabric that a minimum of one instance should always be associated with the profile in the fabric environment and also that a maximum of 3 instances can be associated at any given time.
• Web-Console
• Navigate to the Profiles page on the Hawt-io screen.
• Select the profile to which you want to add the requirements.
• Click the Add-Requirements link to the right of the screen.

• Provide the minimum and Maximum number of hosts to be associated with the profile.

Issues

Once the autoscaler is defined on the root container , the requirements on the profile will be auto-provisioned by the fabric environment . 

• But if for any reason the child containers are brought down , fabric will automatically create a new instance , this might prove to be a problem in case the child containers is being brought down for maintenance.
• If a container is down and is associated to a minimum requirements on a profile , the fabric will spin up a new container but will not attempt to start the shut-down container.

Issue With ObjectMessages 6.2.1 Jboss Fuse and AMQ

If you were working with Jboss Fuse 6.2.0 and migrated to 6.2.1 patch and working with Serialized Java Objects as messages in to AMQ , this post might be of interest to you.
I had a piece of code which was working fine in 6.2.0 version , the code was to simply drop a Java Serialized object in to AMQ  and then read it back.The class is as below

 package com.sundar.verify;  
 import java.io.Serializable;  
 public class MyBean implements Serializable{  
      /**  
       *   
       */  
      private static final long serialVersionUID = 1L;  
      private String id;  
      private String textData;  
      public String getId() {  
           return id;  
      }  
      public void setId(String id) {  
           this.id = id;  
      }  
      public String getTextData() {  
           return textData;  
      }  
      public void setTextData(String textData) {  
           this.textData = textData;  
      }  
 }  

with a camel route

 <camelContext xmlns="http://camel.apache.org/schema/spring">  
           <route>  
                <from uri="timer:foo?repeatCount=1&amp;delay=10000" />  
                <process ref="mybean" />  
                <to uri="activemq:personnel.records" />  
           </route>  
           <route>  
                <from uri="activemq:personnel.records" />  
                <log message="Body is ${body}" />  
           </route>  
 </camelContext>  

This was obsolutely working fine with Fuse 6.2.0 , but when the same code was migrated to 6.2.1 , I suddenly started to notice that the below exception was thrown

 Caused by: javax.jms.JMSException: Failed to build body from content. Serializable class not available to broker. Reason: java.lang.ClassNotFoundException: Forbidden class com.sundar.verify.MyBean! This class is not allowed to be serialized. Add package with 'org.apache.activemq.SERIALIZABLE_PACKAGES' system property.  
      at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36)  
      at org.apache.activemq.command.ActiveMQObjectMessage.getObject(ActiveMQObjectMessage.java:193)  
      at org.apache.camel.component.jms.JmsBinding.extractBodyFromJms(JmsBinding.java:126)  
      ... 23 more  

To prevent this from happening and for the deserilazation to work propertly add the below system property on the client side as below

-Dorg.apache.activemq.SERIALIZABLE_PACKAGES="java.lang,java.util,org.apache.activemq,org.fusesource.hawtbuf,
com.thoughtworks.xstream.mapper,<your-package-names>"  

and for the impatient

-Dorg.apache.activemq.SERIALIZABLE_PACKAGES="*"

In case if you are running the consumer on a fabric container , set the JVM props on the container as in the image below

Fabric Deployment Fuse

Deployments with Fuse And Fabric are very easy , there is a lot of automation available with in fabric which makes writing of scripts to manage devops a redundant process. In this post I am going explore two different ways of pushing changes to Jboss Fuse.

The versions of the products I am using are

1. Fuse 6.2.1 (bom version : 6.2.1.redhat-084). 
2. Fabric8 1.2.0.redhat-621084 (fabric maven plugin version : 1.2.0.redhat-133).

Using Fuse Profile Git - Manually

• Once Fuse Fabric Environment is setup on your local or server. 
• Go to http://<serverip>:8181. 
• Log in to Hawtio using your username and password 
• Navigate to the Containers , click on the root container navigate to Urls tab and copy the Git Url , it should look like "http://admin:admin@smunirAT-OSX:8181/git/fabric”. 
• Open your eclipse clone the repository , provide the username and password if prompted. 
• Create a new branch with a different version number
• Navigate to the working directory on the Git repo cloned , and import the fabric directory as general project to eclipse .
• All the profiles which are now the part fo the branch you checked out from git should now appear in eclipse.

• Edit the file io.fabric8.agent.properties file and change the repositories , features and bundles that you need to add / remove /modify on the profile

 #To add a repository to the profile use the below keyword   
 repository.  
 <repository-name>=mvn\:groupid/<artifactid/version/type/classifer  
   
 #To add a feature to a profile use the below keyword features  
 feature.<feature-name>=<feature-name>  
   
 #To add a bundle to a profile use the below keyword  
 bundle.mvn\:groupid/artifactid/version= mvn\: groupid/artifactid/version  
   

• Commit to the git repo and push to the remote repositories

Video for Demo Project




Look for The automated part of the above ....in my next post

Jmeter Jms Publisher with discovery url Java Sampler

Add the below Jars to lib/ext directory to the Jmeter installation

1. activemq-all-5.11.1.jar
2. fabric-api-1.2.0.redhat-133.jar
3. fabric-groups-1.2.0.redhat-133.jar
4. fabric-utils-1.2.0.redhat-133.jar
5. fabric-zookeeper-1.2.0.redhat-133.jar
6. guava-17.0.jar
7. jackson-annotations-2.4.3.jar
8. jackson-core-2.4.3.jar
9. jackson-databind-2.4.3.jar
10. mq-discovery-1.2.0.redhat-133.jar
11. mq-fabric-1.2.0.redhat-133.jar

Extend the class JavaSamplerClient.

Add arguments to the getDefaultParameters() method

Set the system properties zookeeper.url and zookeeper.password in setupTest() method.

Write the test to connect and send messages to the activemq broker.

For the complete code please click here

When fabric is used to create network brokers fabric assigns free ports so that they will not all run on default port settings , also this provides the flexibility to the users to add more brokers in to the mix later on or remove them when the load becomes lesser.

In this case the better option to connect to these network of brokers instead of using a fail over with all the ip's of the brokers or connecting individual brokers would be to use the discovery URL so to let the fabric manage the details.

Based on the below post from
https://github.com/FuseByExample/external-mq-fabric-client

There are three ways this can be done from Jmeter.Adding the System properties zookeeper.url and zookeeper.password while starting the Jmeter.

Editing the Jmeter.sh and adding the JVM Parameters

1. Edit jmeter.sh.
2. Append the below parameter to the JMETER_OPTS
   
   "-Dzookeeper.url=<zookeeperurl>  -Dzookeeper.password=<zookeeperpassword>".
3. Add the below jars to the lib/ext directory.
   1. activemq-all-5.11.1.jar
   2. fabric-api-1.2.0.redhat-133.jar
   3. fabric-groups-1.2.0.redhat-133.jar
   4. fabric-utils-1.2.0.redhat-133.jar
   5. fabric-zookeeper-1.2.0.redhat-133.jar
   6. guava-17.0.jar
   7. jackson-annotations-2.4.3.jar
   8. jackson-core-2.4.3.jar
   9. jackson-databind-2.4.3.jar
   10. mq-discovery-1.2.0.redhat-133.jar
   11. mq-fabric-1.2.0.redhat-133.jar
4. Run the script jmeter.sh and continue with the JMS publisher and JMS subsriber tests
5. The sample parameters are as below
   1. intialContextFactory=org.apache.activemq.jndi.ActiveMQInitialContextFactory.
   2. providerUrl=discovery:(fabric:<broker-group-name>).
   3. connection factory =ConnectionFactory.
   4. Destination = dynamicQueues/FOO.BAR.

Customizing the JMSPublisher Sampler with GuiControls

1. Copy the same jars from above step to lib/ext directory of Jmeter installation.
2. Write custom code to add a new sampler JMS Activemq Discovery Publisher.I went along the lines of JMSPublisherGui and PublisherSampler from the JMS protocol code , most of the code is copied from them with necessary modifications.
3. Create a class DiscoveryActivemqPublisherGui that extends AbstractSamplerGui and implements ChangeListener
4. Create a class DiscoveryPublisherSampler  which extends
   BaseJMSSampler  and implements TestStateListener
   to write the actual Sampler code.
5. In the initClient method of DiscoveryPublisherSampler write code to set the zookeeper properties to the system variables.
6. Copy the core part of the source code of Jmeter from here.
7. Add the property jms_discovery_publisher= JMS Activemq Discovery Publisher to messages.properties. This will be the label of the Sampler.
8. Use the pom.xml below to compile the code.

Replace the ApacheJMeter_core.jar with the jar obtained above.

For the complete code please click here

Camel Writing a TypeConverter

Using Type Converter Registry and Java DSL

Write your Custom Type Converter by extending the org.apache.camel.support.TypeConverterSupport
The Below Listing provides you a sample of how to do that , this example takes a hypothetical situation where i want to convert a array of objects to a String , nonsensical but nevertheless an example.

package com.sundar.camel.converters; import org.apache.camel.Exchange; import org.apache.camel.TypeConversionException; import org.apache.camel.support.TypeConverterSupport; public class MyConverter extends TypeConverterSupport { @Override public T convertTo(Class arg0, Exchange arg1, Object arg2) throws TypeConversionException { StringBuffer result = new StringBuffer(); Object[] arry = (Object[]) arg2; for (Object arr1 : arry) { if (arr1 != null) { result.append(arr1.toString()); } else { result.append(""); } result.append(","); } String strResult = result.toString(); strResult = strResult.substring(0, strResult.length() - 1); return (T) (strResult + "\n"); } }

Add the TypeConverter to the camel context as shown below

context.getTypeConverterRegistry().addTypeConverter(Object[].class, String.class, new TypeConverterSupport());

Auto Discovery of Type Converters

Write your Type Converter by using annotations as below

import org.apache.camel.Converter; import com.sundar.camel.db.converter; @Converter public class StringConverter { @Converter public String convertTo(<AnyType> arr){ StringBuffer result = new StringBuffer(); return arr.toString(); } }

The above type of implementation is dependent on AnnotationTypeConverterLoader which searches the class-path of your project for a file called TypeConverter (META-INF/services/org/apache/camel/

• Create the file and add the FQN of the type converter we just wrote
  

  com.sundar.camel.converters.StringConverter

• We can also choose to provide the package name of the class and allow Camel to scan the entire package for the TypeConverters
  

  com.sundar.camel.converters

You can use the discovery for the Listing one as well instead of the adding the code and the Type Converter will be added at load time instead of runtime

Now these type Converters can be used as it would like any other converter in Camel
The first Converter would be used as below
<convertBodyToType type="com.sundar.camel.converters.MyConverter"/>
Camel would then invoke the MyConverter to convert the message body
The Second Converter would be used as below
<convertBodyToType type="java.lang.String"/>

Camel would like at the conversions startegies and when the Object Type of the Message
Body matches the Object in the convertTo method , Camel will invoke the StringConverter for the convertsion.