Hadoop as a Batch Job using PBS

During my previous data analyses using Hadoop and CGL-MapReduce I had to use the compute resources accessible via a job queue. For this purpose I used the Quarry cluster @ Indiana University which support batch job submissions via Portable Batch System(PBS). I contacted one of the system administrators of the Quarry (George Wm Turner) and he point me to the Hadoop On Demand (HOD) project of Apache which mainly try to solve the problem that I am facing.

We gave HOD a serious try but could not get it working the way we wanted. What we tried was to install HOD in set of nodes and let users to use it on demand via a job queue. This option simply did not work for multiple users, since the configurations options for the file system overlap between the users causing only one user to use Hadoop at a given time.

With this situation, I decided to give it a try to start Hadoop dynamically using PBS. The task the script should perform is as follows.

1. Identify the master node
2. Identify the slave nodes
3. Update $HADOOP_HOME/conf/masters and $HADOOP_HOME/conf/slaves files
4. UPdate the $HADOOP_HOME/conf/hadoop-site.xml
5. Cleanup any hadoop file system specific directories created in previous runs
6. Format a new Hadoop Distribtued File System (HDFS)
7. Start Hadoop daemons
8. Execute the map-reduce computation
9. Stop Daemons
10. Stop HDFS

Although the list is bit long, most of the tasks are straigt forward to perfom in a shell script.

The first problem I face is finding the IP addresses of the nodes. PBS passes this information via the variable PBS_NODEFILE. However, in a multi-core and multi-processor systems the PBS_NODEFILE contains multiple entries of the same node depending on the numebr of processors we requested in each node. So I have to find the "SET" of IP addresses eliminating the duplicates. Then have to update the configuration files depending on this information. So I decided to use a Java program to do the job (with my shell script knowledge I could not find an easy way to do a "SET" operation)

Once I have this simple Java file to perform the steps 1 -4 the rest is straightforward. Here is my PBS script and I the link after the script will show you the simple Java program which update the configuration files.

----------------------------------------------------------
#!/bin/bash
#PBS -l nodes=5:ppn=8
#PBS -l walltime=01:00:00
#PBS -N hdhep
#PBS -q hod

java -cp ~/hadoop_root/bin HadoopConfig $PBS_NODEFILE ~/hadoop-0.17.0/conf

for line in `cat $PBS_NODEFILE`;do
echo $line
ssh $line rm -rf /tmp/hadoop*
done

var=`head -1 $PBS_NODEFILE`
echo $var

ssh -x $var ~/hadoop-0.17.0/bin/hadoop namenode -format
ssh -x $var ~/hadoop-0.17.0/bin/start-dfs.sh
sleep 60
ssh -x $var ~/hadoop-0.17.0/bin/start-mapred.sh
sleep 60
ssh -x $var ~/hadoop-0.17.0/bin/hadoop jar ~/hadoop-0.17.0/hep.jar hep ~/hadoop_root /N/dc/scratch/jaliya/alldata 40 2
ssh -x $var ~/hadoop-0.17.0/bin/stop-mapred.sh
ssh -x $var ~/hadoop-0.17.0/bin/stop-dfs.sh
-------------------------------------------------------------------
[HadoopConfig.java]

As you can see in the script, I use ssh to log into each node of the cluster and perform the cleaning up of the HDFS directories. Then I use SSH to login to the master node to start the Hadoop daemons.

Next comes the actual execution of the data analysis task, which I have coded in the hep.jar.
After the MapReduce computation is over the rest of the commands simply stop the daemons.

This is method has more flexibility to the user and requires no changes for the batch job scheduling system. It also serves as an easy opti

Save Now

on when the number of MapReduce computations are smaller than the other batch jobs.

However, if the cluster is dedicated to run MapReduce computations, then everybody starting and stopping a HDFS file system does not make sense. Ideally the Hadoop should be started by the adminitrators and then the users should be allowed to simply execute the MapReduce computations on it.

Your comments are welcome!

High Energy Physics Data Analysis Using Hadoop and CGL-MapReduce

After the previous set of tests wtih parallel Kmeans clusting using CGL-MapReduce, Hadoop and MPI I shift the direction of testing to another set of tests. This time, the test is to process large number (and volume of) High Energy Physics data files and produce a histogram of interesting events. The amount of data that needs to be processed is 1 terabyte.

Converting this data analysis into a MapReduce version is straigt forward. First the data is split into managable chunks and each map task process some of these chunks and produce histograms of interested events. Reduce tasks merge the resulting histograms producing more concentrated histograms. Finally a merge operation combine all the histograms produced by the reduce tasks.

I performed the above test by incresing the data size on a fixed set of computing nodes using both CGL-MapReduce and Hadoop. To see the scalability of the MapReduce approach and the scalability of the two MapReduce implementations, I performed another test by fixing the amount of data to 100GB and varying the number of compute nodes used. Figure 1 and Figure 2 shows my findings.

Figure 1. HEP data analysis, execution time vs. the volume of data (fixed compute resources)

Figure 2. Total time vs. the number of compute nodes (fixed data)

Hadoop and CGL-MapReduce both show similar performance. The amount of data accessed in each analysis is extremely large and hence the performance is limited by the I/O bandwidth of a given node rather than the total processor cores. The overhead induced by the MapReduce implementations has negligible effect on the overall computation.

The results in Figure 2 shows the scalability of the MapReduce technique and the two implementations. It also shows how the performance increase obtained by the parallelism diminshes after a certain number of computation node for this particular data set.

July 2nd Report

CGL MapReduce, Hadoop and MPI

Last few weeks I was busy implementing CGL Map Reduce a streaming based map reduce implementation that uses a content dissemination network for all its communication. Our main objective behind this implementation is to avoid the overhead imposed by the technique adopted by both Google and Hadoop in their map-reduce implementations, that is communicate data via files.

Instead of communicating the data between the map and reduce tasks via files, we use NaradaBrokering's publish/subscribe messaging for the data transfer. In most of the map-reduce use cases the output of the map task is significantly smaller than the size of the input data. In addition, the use of a file system based data communication mechanism is prohibitively slow for Iterative map-reduce tasks such as clustering algorithms. These observations motivates us in implementing the CGL Map Reduce.

Following two graphs compares the performance of CGL MapReduce with other parallization techniques used for SPMD programs. For this benchmark we use kmeans algorithm to cluster a collection of 2D data points.

First we compare the performance of CGL MapReduce with Hadoop and MPI. We increase the size of the data set from 100000 points to 40 million and measured the total time for the computation under different implementations. MPI program was the fastest of the three while CGL MapReduce shows very close performance for large data sets. However, Hadoop's timing is almost 30 times longer than the CGL MapReduce and the MPI.

Figure 1. Performance of CGL MapReduce vs. Hadoop vs. MPI

Next, we performed the similar computation on a Single multi-core machine to see the effect of various parallelization techniques for this type of computations. Java threads, MPI and CGL MapReduce shows the converging results for large data sizes. In this case the main limitation factor is the memory access and hence the technique with minimum overhead to the to memory wins the battle. In our experiment the MPI performed the fastest and the Java threads performed second while CGL MapReduce is little behind Java threads. Again, Hadoop is about 30 times slower than any of the other programs.
MPI program achieves its performance from its C++ roots. Java threads is faster than CGL MapReduce simply due to the additional overheads in the map reduce implementation. Hadoop's slowness is due to its overhead in creating and retrieving files for the communication.

Figure 2. Performance of CGL MapReduce vs. Hadoop vs. MPI vs. Java Threads

More analyses will follow soon.

May 21st Report - CTS Conference

Last two weeks, i was preparing for the cts 2008 conference. I had to prepare for my talk and the demonstration. I had to struggle with the new Dell laptop to get the demo working, simply because of the incompatibilities in the software and the necessary drivers. After tweaking some configurations, I was able to install Fedora 7 and ROOT. However, the demo started giving some unpredictable behaviors.

The conference went well, did my talk and did the demonstration as well, the demo crashes few times though.

During the conference, the power went off for the Irvine area and we had to stay without power for almost 16 hours. The organization committee worked hard to get the conference going with some emergency power, and the speakers had to shout the audience since we did not have power for the audio equipments.

After all, it is a very fruitful experience for me. My first talk in a large conference. The keynote speeches and the panel discussions brought a lot of insight into the future of the Collaborative Technologies.

Here are the slides for my presentation.

A ROOT bug when using DLLs

I have been trying to convert the ROOT & C++ program that extends the NaradaBrokering's publish/subscribe functionalities to ROOT users to Windows. I was able to convert the pure C++ part of it and the DLL works fine in Windows.

Then I tried to use the wrappers for ROOT with the generated dictionary. Code generation and the compilation all worked as expected but when I try the DLL the ROOT crashes printing garbage characters.

I track down the problem for two days and was able to reduce the problem into its minimum form.
It simply boils down to a parameter passing problem related to string values.

I then ask the question from the ROOT Talks and one of the ROOT devs, Bertrand, helped me to track down it further.

Finally, the resolution is a bug in ROOT 5.19.02. From my experiments, I know that it was also present in ROOT 5.18 as well.

Here is the full resolution in the ROOT Talk.
http://root.cern.ch/phpBB2/viewtopic.php?p=27702#27702

According to Bertrand, there will be a new release tomorrow and the bug seemed to be fixed in this release.

Hope it will fix my problem and also no more bugs in my way :))

May 7th Report

Posters for the CTS Conference

I have created two posters for the CTS 2008 conference. After a week of crash learning Adobe Illustrator I was able to create them in the way I need :)

Here are the two posters.

Hadoop Presentation

Today I did a small presentation on Apache Hadoop

I went though the documentation they have on HDFS, Map-reduce framework, and the Streaming API. I also had to go through the code to understand some of the functionalities in the framework. So far my idea is that the framework is bit more biased towards the text oriented computations. Probably because the initial computations that they use map-reduce are mainly centered on processing large collection of documents (specifically web pages)

Here is my presentation

April 9th Report

Platform shift for the NBC++ Bridge (Linux to Windows)

The HEP solution is taking a new turn by moving on to CherryPy and Windows. This impose a new requirement for the NaradaBrokering's C++ bridge I wrote.
Initially I did the development in Linux platform using g++ compiler. With the new requirement, I had to compile this for Windows based platforms.

Dr. Julian Bunn gave a big help by converting the C++ bridge to a DLL for windows. After doing some debugging, I was able to get NBC++ working on Windows. Another dimension for NaradaBrokering users.

Since it is hard to maintain two code repositories (one for Windows and one for Linux), I decided to merge the two source code repositories. After covering the differences with pre-compiler directives I was able to get the same code working in both Windows and Linux.

Have to finish the documentation and then I can release the new version of NBC++ soon.

February 27th Report

Writing Papers

The paper, A Collaborative Framework for Scientific Data Analysis and Visualization, I submitted to CTS2008 got accepted. However, reviewers pointed out that there are some grammatical errors in my writing. I had to correct these errors within a week because of the final submission deadline. The following section highlights my experience with the above.

Dr. Shrideep Pallickara pointed me to two books.
1. The Elements of Style by William Strunk Jr. and E.B. White.
2. The Chicago Manual of Style by University of Chicago Press Staff

I found both books in the bookstore and noticed that the second one is more suitable as a reference. The first one is a really nice book for my situation. It is very small book (~100 pages) but it has lot of grammatical styles with examples in it.

I was able to fix lot of grammatical errors present in my initial write up with the help of the above book.

Still I was not 100% sure about my corrections, and decided to search for a proof read service. I found many online services, which does proof reading. However, all of them charge very high amounts for quick jobs.

Lucy Buttersbry, the secretary of our department, pointed me to Writing Tutorial Services in Ballantine Hall 206 (855-6738). If you are an IU student, this is a very good service. They will not proof read your papers, but will help you to identify the common errors you made by going through the paper.

I showed up my revised paper to one of the instructors and he showed me few more common errors that I had in my writing.

After all these steps and few more review cycles I was able to come up with the final version of the paper.

Formatting the Paper

I did the initial writing on Microsoft Word and converted it to pdf before submitting. After converting the document to pdf, I noticed that the column width(for two column pages) of the pdf document is smaller than the column width expected by the conference format sheet.

I decided to give it a try with the Latex format sheet that they provide. After copying few pages I noticed that the column width is correct and also the length of the paper is slightly reduced when formatted using Latex. Of course, the neatness is superb as well.

At the beginning I found it bit hard to insert figures, but it is simply a matter of finding the (right) easy method of doing it. It is very easy to insert images as pdf files. So the only change I had to do was to convert the png images I had into pdf files. I could easily do that using the CutePDF writer. The following latex section shows how I include figures.

\begin{figure}[h]
\begin{center}
\includegraphics[width=0.46\textwidth]{D:/Academic/Ph.D/CTS2008/architecture2.pdf}
\textbf{\caption{\centering{Architecture of the Proposed Collaborative Data Analysis Framework}}
\label{fig:arch}}
\end{center}
\end{figure}

One final thought. If you have time, better to format the paper using Latex as it save you space as well as provide a very neat paper.

February 13th Report

Demonstration to Prof Malcolm Atkinson.

Prof Malcolm Atkinson is the Director of The e-Science Institute and e-Science Envoy National e-Science Centre and was visiting our lab on 4th of February. The lab has organized a demo session so that most students can show their work to him.

When I got the mail regarding the demo, I was in the middle of modifying the ROOT-NB-Clarens application(my research prototype) to add new features. After two long days I was able to get it working and today we did the demo successfully.

January 30th Report

Two papers were due during these and had to struggle with time to get those two papers completed before the deadline.
Followings are the conferences:

1. The 2008 International Symposium on Collaborative Technologies and Systems (CTS 2008)
2. 2008 IPDPS TCPP PhD Forum

Dr. Shrideep and Prof. Fox helped me a lot in refining my ideas and correcting presentation errors. Finally I was able to submit the two papers before the deadline.

I also created a research page which highlights the motivation , goals and the proposed solution of my research. I looking forward to maintain that web site throughout my Ph.D. research.

January 16th Report

Root Client Supports Shared Eventing and Shared Display Type Collaborations.

So far the HEP Data Analysis Client that we have uses a shared event model for collaboration. All the clients perform the fitting and merging of histograms received from servers. This is a very useful feature if different collaborative clients need to "fit" different models to the data received to them. However, if the same model is used by all the clients, then a shared display type collaboration would be the right solution.

I added a feature to the client so that it publishes its current histogram as an image (after fitting and merging) to a topic using the NB's C++ client. I also developed a separate program to simply subscribe to a topic and display the images received over the pub/sub communication channel. This program is very lightweight as its task is merely showing images in a canvas as an when they are received.

This implementation enables the shared display type collaboration among the participating clients to an experiment. Physicists who just need to see the results of an experiment can simply use the shared display client.

Currently all the clients subscribed to a particular topic will receive the histogram images. However, with the introduction of the "agents" , which keep track of on going experiments, to the system these settings can be controlled.

December 26th Report

Scalability of the Rootlet Architecture:

Last two weeks I was working on improving the HEP(High Energy Physics) data processing implementation, so that I can do a benchmark on the scalability of the proposed architecture. As the first step, I was able to benchmark the Naradabrokering's C++ Client that I wrote. The following graph compares the performance of Naradaborkering's Java Client vs. C++ Client.


The graph measures the time for two hops (in milliseconds) for various message sizes. The reason for the step wise increase that the Java client demonstrates is mainly the buffer allocation strategy in Java sockets. During the benchmark a message rate of approximately 50 messages per seconds was maintained.

Next, I measured the time for two hops for a 100KB message with increasing message rates. The results shows that the both Java and C++ implementations show stable performance upto the measured 1000 messages per second message rate. According to the results, the C++ Client performs better than the Java clients for higher message rates. (Please see the graph below)Next Step:
The next task is to measure the scalability of the HEP data processing implementation as a whole. For this I am trying to process large amount of HEP data by increasing the number of processing nodes to process the same amount of data so that we can gain performance improvements by splitting the computation task among multiple processing entities.

MapReduce:

Prof. Fox pointed me to few interesting papers(listed below) which discuss on a technique to parallelize large data processing tasks, named MapReduce, which has its roots in functional programming. Right now I am reading the papers and was simply amazed by the similarity of the work we have done so far the and technique described by these papers:

J. Dean and S. Ghemawat, “Mapreduce: Simplified data processing
on large clusters,” in OSDI’04: Sixth Symposium on Operating System
Design and Implementation, December 2004.

R. Pike, S. Dorward, R. Griesemer, and S. Quinlan, “Interpreting the
data: Parallel analysis with sawzall,” Scientific Programming Journal
Special Issue on Grids and Worldwide Computing Programming Models
and Infrastructure, vol. 13, no. 4, pp. 227–298, 2005.

M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, “Dryad:
Distributed data-parallel programs from sequential building blocks,” in
European Conference on Computer Systems (EuroSys), March 2007.

H. chih Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker, “Map-reducemerge:
Simplified relational data processing on large clusters,” in Proc.
SIGMOD, 2007.

Hope to discuss them more in my next blog.

Decmeber 12th Report

TCSC Symposium Proposal:

After SC07 my main target was to write a paper for the above symposium. According to their website;
"The IEEE TCSC Doctoral Symposium provides a forum for students in the area of Scalable Computing to obtain feedback on their dissertation topics and advice on initiating a research career."

I was able to draft a proposal documentation and then with lot of help from Prof. Fox and Dr. Shrideep we were able to submit it before the deadline.

I learnt a lot regarding writing papers especially in presenting ideas. Coming from the programming background, I always tend to go into details straight away. Sherideep helped me to correct this in the paper.

The paper present our plans on designing a "Scalable Framework for Collaborative Analysis of Sceintific Data" especially for data with the "composition" property. That is, the data analysis task can be broken down to set of sub analyses which can be executed concurrently and merge or combine the results of these sub analyses to form the final results.

November 28th Report

After month of silence:
During the last month I was completely engaged in getting the ROOT client working with Naradabrokering and Clarens so that the Physicist can submit and monitor analysis jobs collaboratively. It was not just another implementation problem, but a implementation+integration task which requires different hetrogenous components to work together. The user case that we tried to achieve.

• A Physicist identify a dataset from a partial physics experiment
• He then write an analysis script based on some analysis criteria using ROOT language and test it using a sample data file in his computer.
• Now he needs to execute this analysis on all the data files available for a particular experiment.
• While the jobs are getting processed, he should be able to monitor the results of each analysis sub task, which is a histogram.
• The client program that needs to be developed should be able to display and merge the resulting histograms in real time.
• Also, any other physicist who would like to see the result of the analysis as an when it is happening, should be able to connect to the same experiment and see the results getting merged one by one in his Client Software.

With a month of work I was able to implement a Client software written in ROOT language that can achieve all of the above requirements. We were able to show this demonstration during the Supercomputing Conference 07 in Reno Nevada. The following image shows the software while executing an analysis on files located in three different servers.

Some explanation about the software:

• Main canvas shows the histogram of the results. All the histograms generated at each analysis sub task is merged and displayed to the user
• Panel at the top right; shows the connected server. In this example, it has been connected to three Clarens servers running in three different machines.
• Panel below that; shows the available data files at each server. This panel also shows the status of each file, whether it has been processed or not by changing color. "Grey" color indicates that the file has not yet been processed, "Red" color indicates that the resulting histogram for that file has been received and in the process of merging it with the available results so far, and the "Blue" color indicates that the file has been processed and the resulting histogram has been merged with the existing results.

Few Implementation Details:
The GUI is completely written in the interpreted language provided by the ROOT framework.
It uses a C++ bridge for Naradabrokering and C++ client library for Clarens server internally.
It also uses a python script to submit analysis tasks to multiple Rootlets (A concept similar to Servlet) concurrently. This is especially required because the interpreted ROOT language does not support multi-threading.

October 3rd Report

Had a meeting with Prof. Geoffrey relating to the Ph.D. topic that I should select. He advice me to find more use cases of the data analysis tasks that are similar to the Particle Physics data analysis that we are doing using Clarens and ROOT.

The data analysis tasks that we handled has the following characteristic.

• Data is in large files and these files are distributed across the globe.
• One or more analysis technique(in our case, one analysis script) can be applied to all the data to identify patterns.
• The outcome of an analysis of a single file is a histogram.
• These outcomes(histograms) can be merged to produce the final results.

So far I have found one strong use case of this nature and that is;
Astronomical Image Processing - mainly for identifying features in astronomical images.

There are few candidate areas that I found interesting and they are;
Analysis of Earthquake Data
Microarray Analysis for Genes
Pattern Matching in Financial Data

Currently I am reading to find out the exact data analysis requirements of these fields. The target is to find more use cases for "Distributed Composable Data Analysis"

September 19th Report

Conrad tested the demo from CERN and it worked well. So, now I can focus on the next step of the project.
Conrad also sent me a link to more root data files so as the next step I will test the demo with those new root data files. The first demo only uses a single rootlet and the reason for this is mainly the way how the Clarens client is written. Each analysis request is processed synchronously and hence the client send requests one by one to the server for each root data file to be analyzed.

As the next step of the project, I am planning to run the client in with multiple processes and allow it to create multiple rootlets so that the analysis can be performed simultaneously utilizing the full cpu power. Hope to get the results soon.

September 5th Report

Demo is ready.

ROOT application development supports both interpreted code and also compiled code. Interpreted code (Mainly the analysis code written by physicist) can use compiled shared libraries including ROOT provide ones and also other user written libraries. Interpretation helps the user to debug and fix code easily and write the necessary analysis as a function.

As explained in my previous blog (August 22nd Report) the plan is to let the rootlet to publish the location of the generated histogram files to the subscribed clients. Clients who receive those notifications then use these files to update their results.

As the second phase, I developed a ROOT compliant wrapper classes for the C++ client of Naradabrokering. These allow ROOT clients to utilize the publish/subscribe capabilities of Naradabrokering.
The steps for developing the ROOT compliant classes are explained in the following tutorial.
Part1, part2 and part3

I was able to get the publishing of messages working from interpreted code and thought that the subscription would work in the same manner. After few days of trying I found that I was wrong.
To get the subscription to work, the interpreted code should pass a function pointer as the callback to the compiled code and on the reception of a notification , the compiled code should call this callback (which is in the interpreted code). Passing a function pointer to the compiled code is easy and straight forward, however, when the compiled code try to call that pointer I got an error *****Segmentation Violation*********

This happens mainly because of the limitation of the ROOT interpreter in resolving function pointers across the interpreted/compiled code boundaries. After few days of searching and querying Conrad, we decided to ask the question on the ROOT mailing list. They replied really soon and helped us to solve the problem. The solution came in a way of a reflection type call interface for calling interpreted code from the compiled code supported by ROOT.
Here is the mail thread.
http://root.cern.ch/phpBB2/viewtopic.php?t=5408

So with that help, I was able to get the demonstration working and it is so nice to see how it is working. Results of the remote analysis triggers users who are subscribed for notifications and their histograms get updated with remote results.

Somethings are too good to be true!

August 22nd Report

Last week Conrad helped us in setting up a Clarens server in gridfarm003 and after resolving few issues with our certificates I was able to use it.

Usage Scenario : Big Picture

• The user writes a Client code in C++ which utilizes services of the clarens server. Let's call this ClientCode.C
• She also has written the analysis code for root data. Let's call these files Analysis.C and Analysis.H .
• She then executes the ClientCode.C using the C++ interpreter provided by the ROOT. (ROOT has a built in C++ interpreter)
• ClientCode.C uses the built in ROOT libraries to locate data files in the server and also to upload the above two files to the clarens server.
• After discovering (polling for files) root data files, ClientCode.C send request/requests to the Clarens server for creating rootlet/rootlets to execute the analysis code that it uploaded.
• For every rootlet request Clarens server creates a wrapper script for rootlet and executes it with the input and output files. This wrapper script is called rootlet_wrappper.sh
• Finally the ClientCode.C poll for output files and display the results of the analysis in a histogram generated at the user's machine.

Incorporating Naradabrokeing

As the first step, I changed the rootlet_wrapper.sh to publish a message after finishing the analysis using the nbclient program we wrote using C++. This works fine and we can eliminate the polling requirement of the ClientCode.C to find the results.

Next task is to incorporate the subscriber functionality to the visualization part of the ClientCode.C

August 8th Report

Secure Message Transfer Between Java and C++

Scenario:

We are developing an application which require secure message transfer between Naradabrokering (java based messaging substrate) and C++ client application. The communications between the entities uses a custom publish/subscribe messaging protocol to get better performance. (No XML processing)

JDK has a built in support for security features such as certificate handling, encryption and signing. However, to get those functionalities in C++ a separate library needs to be installed. For this we used Openssl (http://www.openssl.org/) To develop applications it is required to have the development files of the openssl and the installation is different according to the underlying operating system. For my machine running Ubuntu 2.6.15-28-386 it is simply;

apt-get install libssl0.9.7
apt-get install libssl-dev

Following sections of shows the code fragments that we can use to encrypt/decrypt messages (bytes) both in Java and C++. The algorithm used for the encryption is AES (http://en.wikipedia.org/wiki/Advanced_Encryption_Standard)

Encryption in JAVA

In java the encryption is handled by the provided javax.crypto.Cipher class. The following code fragment shows the encryption in Java.

byte[] bytesToEncrypt = /*Bytes to be encrypted*/
byte[] encBytes = null; /*Encrypted Bytes*/

/**
* Create a Cipher by specifying the following parameters a. Algorithm
* name - here it is AES */

Cipher aesCipher;
try {
aesCipher = Cipher.getInstance(Constants.AES_ALGO);

aesCipher.init(Cipher.ENCRYPT_MODE, secretKey);
encBytes = aesCipher.doFinal(msg.getBytes());
} catch (Exception e) {
throw new ClarensException(
"Error encrypt message using secret key",e);
}

These bytes are then transferred to the C++ client using socket based communication channel.

Decryption in C++

Openssl provides a set of libraries for handling the decryption and the following utility function shows how we can use those to decrypte the received set of bytes.

bool
SecurityUtil::decryptAES(const unsigned char *in,int inputLength ,unsigned char *out,int &outputLength, string aesKey){

int olen, tlen, n;
EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init (& ctx);
EVP_DecryptInit (& ctx, EVP_aes_128_ecb (), (unsigned char *)aesKey.c_str(), NULL);

olen=0; tlen=0;

if (EVP_DecryptUpdate (& ctx, out, & olen, (const unsigned char*)in,inputLength) != 1)
{
cerr<<"error in decrypt update"< return false;
}

if (EVP_DecryptFinal(& ctx, out + olen, & tlen) != 1)
{
cerr<<"error in decrypt final"< return false;
}

olen += tlen;
outputLength=olen;

EVP_CIPHER_CTX_cleanup (& ctx);
return true;
}

Ok, now let's see the other side of the story, from C++ to Java

Encryption in C++

Again Openssl provides a set of library functions for encryption as well. Following is the utility function for the encryption

bool

SecurityUtil::encryptAES(const unsigned char* in,int
inputLength ,unsigned char *out,int &outputLength, string
aesKey){

int olen, tlen, n;
EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init (& ctx);

EVP_EncryptInit (& ctx, EVP_aes_128_ecb (), (unsigned char
*)aesKey.c_str(), NULL);

if (EVP_EncryptUpdate (& ctx, out, & olen, (const unsigned
char*)in , inputLength) != 1)
{
cerr<<"error in decrypt update"<

if (EVP_EncryptFinal (& ctx, out + olen, &amp;amp;amp;amp;amp;amp; tlen) != 1)
{
cerr<<"error in encrypt final"<

olen+=tlen;
outputLength=olen;

EVP_CIPHER_CTX_cleanup (& ctx);
return true;
}

Decrypting the bytes received from the C++ client in JAVA

Decryption is java is also handled by the javax.crypto.Cipher class and is fairly straight forward. Here is the code segment.

byte[] decBytes = null;

Cipher aesCipher;

try {
aesCipher = Cipher.getInstance(Constants.RSA_ALGO);

aesCipher.init(Cipher.DECRYPT_MODE, prKey);
decBytes = aesCipher.doFinal(msgBytes);
} catch (Exception e) {
throw new ClarensException(
"Error decrypting message using private key ", e);
}

Simple right? The main problem I faced when developing the above application was the lack of documentation on this regard. There are tons of documentation on how to handle encryption/decryption using java but very small number for the same in C++. How about encryption/decryption between Java and C++? I could not find anything in this sort. Openssl has a good documentation on various functions/data structures it offers for encryption/decryption but the main problem is there very limited amount of code examples which shows the exact usage. Followings are some of the resources that I used to come up with this implementation and hope someone will find this helpful.

http://www.openssl.org/docs/
http://www.madboa.com/geek/openssl/#cert-self
http://www.ibm.com/developerworks/linux/library/l-openssl.html
http://www.mail-archive.com/openssl-users@openssl.org/msg40449.html
http://www.mail-archive.com/openssl-users@openssl.org/msg23119.html
http://www.fortrel.net/blog/index.php?title=encryption_java_c&more=1&amp;c=1&tb=1&pb=1
http://www.adp-gmbh.ch/cpp/common/base64.html

Next Blog: Signing and Verifying between Java and C++