Jaliya Ekanayake Tech Report

Azure Platform Appliance

2010-07-16T10:27:00.000-07:00

Microsoft recently announced its next move with Azure "Azure Platform Appliance". The idea is to offer Azure platform as an infrastructure that can be used in private clusters. From my perspective, this is somewhat similar to the idea of private clouds (infrastructure services) that one can deploy using runtimes such as Eucalyptus or Nimbus on local clusters. However, since Azure is not an infrastructure service, it will be more flexible for the users. Unlike pure virtual machines, Azure platform appliance will expose most platform services of Azure as well. In addition, the migration between the private Azure and the public Azure will be seamless as well. Overall, I think the biggset advantage of this approach is the piece of mind that the businesses will have "I am in control of my data and it is local".

Azure Storage Browser in Visual Studio

2010-06-24T16:12:00.000-07:00

Now this is a handy tool.

Hadoop Application Wiki

2010-06-11T14:54:00.001-07:00

A nice collection of Hadoop apps.
http://wiki.apache.org/hadoop/PoweredBy

What is the concept of "static data" in Twister?

2010-03-18T21:31:00.000-07:00

Many iterative applications we analyzed show a common characteristic of operating on two types of data products called static and variable data. Static data (most of the time the largest of the two) is used in each iteration and remain fixed throughout the computation whereas the variable data is the computed results in each iteration and typically consumed in the next iteration in many expectation maximization (EM) type algorithms.

If a set of data is read by the application but do not get changed, then this set of data can be considered "static" in Twister. Matrix blocks, points in clustering, and web graph in pagerank are all examples of static data.

How does Twister differ from MRnet?

2010-03-18T21:27:00.000-07:00

There are many differences between MRnet and MapReduce and then with Twister. I will list some here.

In MapReduce the communication between map and reduce stage of the computation is a graph not a tree. A map can produce (key,value) pairs that may end up in multiple reducers. In MapReduce the framework does not impose any communication topology or connection topology between map and reduce stages of the computation. It is purely the intermediate keys that determine the communication pattern. For example, in an associative and commutative operations such as sum or histogramming, how the intermediate keys are used to distribute intermediate results among the reduce tasks is not that important.However for operations such as sorting, or matrix operations, one can select the intermediate keys in such a way that specific keys goes to specific reduce tasks. Again this is not defined by the network, but the keys and the key selector functions.

Twister uses pub/sub messaging to implement a MapReduce runtime,especially to support iterative MapReduce computations. Similar to other MapReduce runtimes it gives more focus on processing data while maintain data- process affinity. Map and reduce functions in Twister are long running processes providing distinction between static data and variable data. It supports broadcast, scatter type data distributions and reading data via the local disks. I am not sure how the latter two functions can be handled using MRnet.

Comparing MRnet with MapReduce for data processing applications is an interesting analysis one can do though.

Twister: Iterative MapReduce

2010-02-09T15:26:00.000-08:00

MapReduce programming model has simplified the implementations of many data parallel applications. The simplicity of the programming model and the quality of services provided by many implementations of MapReduce attract a lot of enthusiasm among parallel computing communities. From the years of experience in applying MapReduce programming model to various scientific applications we identified a set of extensions to the programming model and improvements to its architecture which will expand the applicability of MapReduce to more classes of applications.

Twister is a lightweight MapReduce runtime we have developed by incorporating these enhancements. We have published several scientific papers [1-5] explaining the key concepts and comparing it with other MapReduce implementations such as Hadoop and DryadLINQ. Today we would like to announce its first release.

Key Features of Twister are:

Distinction on static and variable data
Configurable long running (cacheable) map/reduce tasks
Pub/sub messaging based communication/data transfers
Combine phase to collect all reduce outputs
Efficient support for Iterative MapReduce computations (extremely faster than Hadoop or DryadLINQ)
Data access via local disks
Lightweight (5600 lines of code)
Tools to manage data

For more details please visit www.iterativemapreduce.org and let us know your thoughts and experience using Twister.

Thank you,

SALSAHPC Team.

[1]. Jaliya Ekanayake, (Advisor: Geoffrey Fox) Architecture and Performance of Runtime Environments for Data Intensive Scalable Computing, Doctoral Showcase, SuperComputing2009.
[2]. Jaliya Ekanayake, Atilla Soner Balkir, Thilina Gunarathne, Geoffrey Fox, Christophe Poulain, Nelson Araujo, Roger Barga, DryadLINQ for Scientific Analyses, Fifth IEEE International Conference on e-Science (eScience2009), Oxford, UK.
[3]. Jaliya Ekanayake, Xiaohong Qiu, Thilina Gunarathne, Scott Beason, Geoffrey Fox High Performance Parallel Computing with Clouds and Cloud Technologies Technical Report August 25 2009 to appear as Book Chapter.
[4]. Geoffrey Fox, Seung-Hee Bae, Jaliya Ekanayake, Xiaohong Qiu, and Huapeng Yuan, Parallel Data Mining from Multicore to Cloudy Grids, High Performance Computing and Grids workshop, 2008. – An extended version of this paper goes to a book chapter.
[5]. Jaliya Ekanayake, Shrideep Pallickara, Geoffrey Fox, MapReduce for Data Intensive Scientific Analyses, Fourth IEEE International Conference on eScience, 2008, pp.277-284.

Almost MapReduce in 1992

2010-01-03T21:42:00.000-08:00

Paralex: An Environment for Parallel Programming in Distributed Systems (1992)

by Özalp Babaoglu , Lorenzo Alvisi , Alessandro Amoreso , Renzo Davoli , Davoli Luigi , Luigi Alberto Giachini

Just found this paper and read it to the end since I noticed some similarities of what they have proposed in 1992 and the current MapReduce programming model and some of the observations are still true for today as well. I will list few of the observations/assumptions they have made showing the similarity of their work and the current MapReduce programming model.

Large-grain data flow model suitable for high-latency low bandwidth networks
Only by keeping the communication-computation ratio to reasonable levels can we expect reasonable performance from parallel applications in such system. – We noticed a similar thing with performing parallel computing in Cloud infrastructures [paper]
Paralex functions mush be “pure” – no side effects
Node corresponds to computations (functions, procedures, programs) and links indicate flow of typed data – Compare this with Microsoft Dryad’s DAG based programming model.

Dryad paper has referred to their work. - I haven't noticed it before ;)

Some of their performance measures had issues with 16MB data set because the memory they had in one of the machines was only 16 MB. Today we have the luxury of using large memories but our data sets are also grown into petabytes. What they did with NFS is now done in HDFS in Hadoop.

Dynamic Provisioning of Virtual Clusters

2009-12-01T12:15:00.001-08:00

Here I will present the details of the demonstration that we (SALSA team) presented at the Super Computing 09 conference in Portland.

Deploying virtual/bare-system clusters on demand is an emerging requirement in many HPC centers. The tools such as xCAT and MOAB can be used to provide these capabilities on top of a physical hardware infrastructure.

In this demonstration we coupled the idea of provisioning clusters with parallel runtimes. In other words, we showed that people can switch a given set of hardware nodes into different operating systems (either virtual or non-virtual) and run different applications written using various cloud technologies. Specifically we showed that it is possible to provision clusters with Hadoop on both Linux bare-system and virtual machines and Microsoft DryadLINQ on Windows Server 2008.

The following diagram shows the operating system and the software stacks we used in our demonstrations. We were able to get all configurations demonstrated except for Windows HPC on XEN VMs which does not work well due to the non-existence of para-virtualized drivers.

We setup our demonstration in 4 clusters each with 64 CPU cores (8 nodes with 8 CPU cores each). The first 3 clusters were configured with RedHat Linux bare-system, RedHat Linux on XEN, and Windows HPC 2008 operating systems respectively. The last cluster is configured to dynamically switch between any of the above configurations. In Linux (both bare-system and XEN) clusters we ran a Smith Waterman dissimilarity calculation using Hadoop as our demo application while on Windows we ran a DryadLINQ implementation of the same application.

We developed a performance monitoring infrastructure based on pub-sub messaging to collect and summarize CUP and memory utilization of individual clusters and a performance visualization GUI (Thanks Saliya for the nice GUI) as our front end demonstration component. Following two diagrams show the monitoring architecture and the GUI of our demo.
With the 3 static clusters we were able to demonstrate and compare the performance of Hadoop (both on Linux bare-system and XEN) and DryadLINQ to the people. It also provides a way to show the overhead of virtualization as well. The dynamic cluster demonstrated the applicability of the dynamically provisionable virtual/physical clusters with parallel runtimes for scientific research.

It was a wonderful team work involving all the members of the SALSA team and the members of IU UITS. What is more amazing is that we go this very successful demonstration built from the scratch in less than one month.
We will upload a video of the demonstration soon.

Here is a photo of most of the members of our group.
Front row (left to right): Jaliya Ekanakaye, Dr. Judy Qiu, Thilina Gunarathne, Scott Beason, Jong Choi, Saliya Ekanayake, Li Hui.
Second row (left to right) Prof. Geoffrey Fox, Joe Rinkovsky and Jenett Tillotson.

CloudComp09 Presentation

2009-11-01T21:03:00.000-08:00

I shared the slides presented at CloudComp09 via slideshare.

Windows Server 2008 on Xen

2009-11-01T20:41:00.000-08:00

As part of our ongoing research, we started running DryadLINQ applications on a virtual cluster running Windows Server 2008 VMs on top of Xen. Although we noticed acceptable performance degradation with para-virtualized Linux guests (from our previous research), with Windows we noticed extremely higher performance degradations. Also we noticed that whenever we run a DryadLINQ job which has considerable amount of communication, the VM that runs the DryadLINQ job manager crashes. We had an exactly similar experience with Windows VMs provided by GoGrid earlier. However, those VMs had only 1 CPU core where as the VMs we just ran have 8 CPU cores. (Note: This effect has nothing to do with DryadLINQ, but the handling of I/O operations by the guest and the host OSs)

Overall, the full virtualization approach may not seem to work well for parallel applications. Especially for applications with considerable inter-process communication requirements. Following is a brief analysis I wrote.

===========================
If we look at the development of the virtualization technologies, we can see these three virtualization techniques.

Full virtualization -> Para-Virtualization -> Hardware Assisted Virtualization

(e.g. VM Ware Server) e.g. Xen, Hyper-V e.g. Hyper-V, Virtual Iron, VM-Ware Workstation (64 bit)

So far, para-virtualization seems to be the best approach for virtualization to achieve better performance. However, it requires modified guest operating systems. This is the problem for Windows guests, and that is why we cannot run Windows Server 2008 on Xen as para-vritualized guest. According to [1] the hardware assisted approach is still in its early stages and not performing well, but I think this will catch up soon.

Hyper-V coming from Microsoft, may provide better virtualization solutions to Windows guests and currently it supports both para-virtualization and hardware assisted virtualization.

Given these observations and the observations from our tests, currently it does not seem feasible to have one virtualization layer (like Xen) on a cluster and provide both Windows and Linux VMs for parallel computations. (Here I assumed that Linux on Xen is performing better than Linux on Hyper-V, we need to verify this).

However, if we go for a hybrid approach such as using technologies like XCAT to boot Windows and Linux bare-metal host environments, and provide Linux virtualization using Xen and Windows virtualization using Hyper-V, we may be able to utilize best technologies from both worlds.

Then the next thing to figure out is the performance and how we can run Dryad, Hadoop, and MPI on these virtual environments.

Although we did not see this “mythical” 7% virtualization overhead, they are not so bad as well (on private clouds /clusters at least – we saw 15% to 40% performance degradations).

However, we need to figure out ways of handling large data sets in VMs to use Hadoop and Dryad for wide variety of problems. The main motivation of MapReduce is “moving computation to data”. In virtualized environments, we currently store only an image of the VM without data. If the data is large we have to either move them to the local disks of virtual resources before we run computations, or attach blobs/block storage devices to VMs. The first approach does not preserve anything once the virtual resource is rebooted, whereas the second approach adds network links in between the data and computations (moving data). If our problems are not so data intensive, we can simply ignore these aspects for the moment, but I think this is something worth investigating.
==========================
Hope to add more results soon.

Tips for MapReduce with Hadoop

2009-09-17T08:37:00.001-07:00

I found these nice set of tips for fine tuning MapReduce programs using Hadoop from the Cloudera web site.

MSR Internship is over - Going back to IU

2009-09-11T22:16:00.000-07:00

Today I finished my 3 months internship at Microsoft research. It was quite a wonderful experiance for me, and I was able to accomplish most of my internship goals.

At the beginning of my internship I was given the following goals for my internship.
Evaluate the usability of DryadLINQ for scientific analyses
– Develop a series of scientific applications using DryadLINQ
– Compare them with similar MapReduce implementations (E.g. Hadoop)
– Run above DryadLINQ applications on Cloud

During the internship, I developed four DryadLINQ applications and optimized them for performance and also identified several improvements to the current DryadLINQ code base.

I did a detailed performance analysis of the Cap3, HEP, Kmeans applications developed using DryadLINQ comparing them with Hadoop implementations of the same applications. Performance of the pair wise distance calculation application was compared with an MPI implementation of the same application. These findings were all included in the following two papers.
DryadLINQ for Scientific Analyses
Cloud Technologies for Bioinformatics Applicaitons

We (I and my colleague intern –Atilla Balkir) were able to deploy a Windows HPC cluster on GoGrid cloud. I was able to run Cap3 application on Cloud but other applications did not work due to the limitations of the GoGrid infrastructure.

Overall we have the following conclusions regarding DryadLINQ runtime.

We developed six DryadLINQ applications with various computation, communication, and data access requirements
All DryadLINQ applications work, and in many cases perform better than Hadoop
We can definitely use DryadLINQ for scientific analyses
We did not implement (find)
–Applications that can only be implemented using DryadLINQ but not with typical MapReduce
Current release of DryadLINQ has some performance limitations
DryadLINQ hides many aspects of parallel computing from user
Coding is much simpler in DryadLINQ than Hadoop (provided that the performance issues are fixed)
More simplicity comes with less control and sometimes it is hard to fine-tune
We showed that it is possible to run DryadLINQ on Cloud

I got all the necessary support from my mentor (Nelson Araujo), Chirstophe, and the ARTS team @ MSR in accomplishing the objectives of my internship. I would also like to thank Dryad team at Silicon Valley for their dedicated support as well. Last but not least, the support from my advisor (Prof. Geoffrey Fox) and the SALSA team at pervasive technology labs was a tremendous encouragement to me.

Sunday we are planning to head back to Indiana with a two week old baby - Our small miracle - in our hands.

DryadLINQ for Scientific Analyses

2009-09-07T21:04:00.001-07:00

I spent the last 3 months at Microsoft Research as an intern doing research on DryadLINQ. Our goal (myself and a another intern - Atilla Soner Balkir) was to evalute the usability of DryadLINQ for scientific applications.

We selected a series of scientific applications and developed DryadLINQ programs for those applications, and evaluated their performances. We compared the performance of the DryadLINQ applicaitons against Hadoop and in some cases MPI versions of the same applications.

We identified several improvments to DryadLINQ and its software stack, and found workarounds to these inefficienies and was able to run most applicaitons with 100% CPU utilizations.

We compiled a paper including our findings regarding DryadLINQ and submitted it for the eScience09 conference. You can find a draft of this technical paper here.

Hope this will be usefull to some of you who are developing applications using DryadLINQ.

Microsoft released Dryad and DryadLINQ for Academic Use

2009-07-13T11:43:00.001-07:00

http://research.microsoft.com/en-us/collaboration/tools/dryad.aspx

High Performance Parallel Computing with Clouds and Cloud Technologies

2009-06-19T00:33:00.000-07:00

We compiled the latest results/findings of our research as a paper and submitted to CloudComp2009.
Following is the abstract of the paper.

Infrastructure services (Infrastructure-as-a-service), provided by cloud vendors, allow any user to provision a large number of compute instances fairly easily. Whether leased from public clouds or allocated from private clouds, utilizing these virtual resources to perform data/compute intensive analyses requires employing different parallel runtimes to implement such applications. Among many parallelizable problems, most “pleasingly parallel” applications can be performed using MapReduce technologies such as Hadoop, CGL-MapReduce, and Dryad, in a fairly easy manner. However, many scientific applications, which require complex communication patterns, still require optimized runtimes such as MPI. We first discuss large scale data analysis using different MapReduce implementations and then, we present a performance analysis of high performance parallel applications on virtualized resources.

You can find the draft of the paper here.

How to control the number of tasks per node when you run your jobs in Hadoop cluster?

2009-05-14T11:42:00.000-07:00

One of my colleague asked me the following question.

How to control the number of tasks per node when you run your jobs in Hadoop cluster?

We can do this by modifying the hadoop-site.xml. However, the exact xml for the properties are there in hadoop-default.xml.

So here is the method.

Modify the $HADOOP_HOME/conf/hadoop-site.xml and add the following lines.

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>8</value>
</property>

<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>2</value>
</property>

You can find these properties in Hadoop-default.xml, but better not modify them there.
Instead, copy the properties to hadoop-site.xml and change the value. Then the default values will be overridden by the properties in the hadoop-site.xml.

JobConf.setNumMapTasks(). Is to define how many map tasks Hadoop should execute for the entire job. This simply determines the data splitting factor.

With all three parameters we can control the Hadoop's job parallelism better.

High Energy Physics Data Analysis using Microsoft Dryad

2009-05-13T16:35:00.001-07:00

A demo of the Dryad version of the HEP data analysis can be found here.

Classes of MapReduce Applications and Different Runtimes

2009-04-05T09:16:00.000-07:00

In my experiance with MapReduce runtiems and the applications, I have noticed few classes of applications where the users can benifit from the available MapReduce runtimes. Let me list them below with their common characteristics.

1. Pleasingly Parallel Applications.
E.g. Processing a set of bilogy data files using Cap3 program.

This could be the most common class of parallel applications that many users needs. The common characteristic is the application of a function or a program on a collection of data files producing another set of data files. Processing a set of medical images is also another type of similar application.

When using MapReduce runtimes for these type of applications, we can simply use "map only"
mode of the runtimes. For example, in Hadoop we can use 0 reduce tasks as follows.

JobConf jc = new JobConf(getConf(), Cap3Analysis.class);
jc.setNumReduceTasks(numReduceTasks);

I impletented the "map only" support for CGL-MapReduce as well and it greatly increase the usability of the runtime to another large class of parallel applications.

More information about Cap3 and the performance of different runtimes such as Hadoop, Dryad, and CGL-MapReduce can be found in the CGL-MapReduce web page.

This class of applications also suite best for the Azure applications written using Worker Role where a set of workers are assinged to process a computation tasks available in a Queue.

2. Typical MapReduce Applications

E.g. High Energy Physics Data Analysis, where a set of analysis functions are executed on each data file of a collection of data files during the "map" stage producing a collection of intermediate histograms and in the "reduce" stage these histograms are merged together to produce the final histogram.

Histogramming words and Distributed Grep are very common examples used in most tutorials.

This class utilizes complete execution model of MapReduce. The application of the "reduce" stage and the function performed by that stage depend on the "associative" and "commutative" properties of the application itself.

3. Iterative MapReduce Applications.

This is a complex class of applications where we need to apply mutliple stages of MapReduce computations to accomplish the data analysis task. Kmeans clustering is a simple application of this nature, where each iteration of MapReduce cluster a set of data points to a given number of cluster centers. The next iteration takes the previous cluster centers as the input and calculate another set of cluster centers minimizing the difference.

Matrix Multiplication and Multi Dimensional Scaling aretwo more similar applications of this class and many machine learning algorithms also fit to this class of applications.

However, the applicability of the current runtimes such as Hadoop and Dryad for this class of applications is questionable. According to our results shown in the following graph we can see the overhead added by runtimes such as Dryad and Hadoop for such data analysis. Also note how the light weight MapReduce runtime -CGL-MapReduce - and its support for iterative applications have enabled it to produce close results with MPI.

Figure 1. Performance of different parallel run times in executing Kmeans Clustering

Figure 2. Overhead of different parallel run times in executing Kmeans Clustering

Simple PasswordLess SSHTutorial

2009-04-05T09:10:00.000-07:00

I was configuring Xen virtual machines to run MPI programs. Running MPI,Hadoop, or even CGL-MapReduce require password less login between the nodes of a cluster.

It is the same old steps that we need to perform, but found this simple tutorial which list the process very succinctly.

To get MPI to work you also need to add the following line to your bash (or your shell) profile.

export LAMRSH="ssh -x"

Eucalyptus without a DNS

2009-02-27T22:53:00.000-08:00

It has been a while that I last blog relating to my research. I went to my home country (Sri Lanka) for the winter break and came back in mid January. After coming here I just realized that I need a big vacation again :). I had to finish a lot of pending work and now back on track and I can blog again. Still it is 1.56 am in the morning :)

I just got a set of Xen VMs running with Eucalyptus after a few hours of debugging to fix the following warning that I got when I try to ssh between the VM instances.
-----

get_socket_address: getnameinfo 8 failed: Name or service not known
userauth_hostbased: cannot get local ipaddr/name
-----

Actually, it has nothing to do with the VM usage but something I have missed in configuring host based authentication using ssh.

Since it could be helpful to someone else, let me describe the problem and the solution.
I was trying to start a set of VM instances using a Eucalyptus cloud that has been setup in a iDataplex cluster here at Indiana. I was able to follow the guidelines and start the VMs.

Then my goal is to run a set of MPI applications on these VMs. To get MPI working, I need to have password less login between the nodes. As I have previously configured this in the VM image things worked fine but it gave the above warning when I try login between nodes, which then stops MPI daemons from starting.

When I used these VM images with Eucalyptus public cloud every VM instance got assigned a static IP. So, setting up any DNS was not required. This time, however, we use only one static IP for one of the nodes of the VM instances while the rest have dynamic IPs. Also there is no DNS setup for these VM instances. This causes the above warning when I try to login between the nodes using ssh.

Joe Rinkovsky a Unix Systems Specialist @ Indiana explained me the fix. I need to add (IP , name) pairs to /etc/hosts file of the head node of the VM instances. The solution is explained here under "make sure name resolution works" which I haven't noticed till today :).

MapReduce for Data Intensive Scientific Analyses

2008-11-17T11:00:00.000-08:00

Most scientific data analyses comprise analyzing voluminous data collected from various instruments. Efficient parallel/concurrent algorithms and frameworks are the key to meeting the scalability and performance requirements entailed in such scientific data analyses. The recently introduced MapReduce technique has gained a lot of attention from the scientific community for its applicability in large parallel data analyses. Although there are many evaluations of the MapReduce technique using large textual data collections, there have been only a few evaluations for scientific data analyses. The goals of this paper are twofold. First, we present our experience in applying the MapReduce technique for two scientific data analyses: (i) High Energy Physics data analyses; (ii) Kmeans clustering. Second, we present CGL-MapReduce, a stream based MapReduce implementation and compare its performance with Hadoop.

The paper accepted for the eScience 2008.

Multi Dimensional Scaling (MDS) using MapReduce

2008-10-10T12:38:00.000-07:00

As part of my ongoing research on various parallelization techniques, I implemented a MapReduce version of a MDS program. My colleague Seung-Hee Bea has developed the in memory implementation of MDS using C# which runs on a multi-core computers. As the first step in converting that algorithm to a MapReduce implementation, I implemented a single threaded version of the same program in Java (since I am planning to use CGL-MapReduce as my MapReduce runtime). I ran this program to reduce 1024 four dimension (4D) data set to a 3D data set and measured the execution time of the different parts of the program. The result showed that 97.7% of the overall execution time of this program is used for matrix multiplication (both square matrices and matrix and vector multiplications).

This observation motivated me to first implement the matrix multiplication in MapReduce and parallelize the matrix multiplication part of the entire computation using MapReduce technique. I developed both square matrix and matrix and vector multiplication algorithms using CGL-MapReduce and incorporated these to the MDS algorithm.

I tested this MapReduce version of the MDS program using 1024 and 4096 points data sets (4 dimensional data reduced to 3D data) and obtained the following results. The data I used has a predefined structure and the results show the exact structure of the data. In addition the results also tally with the results of the single threaded program. So I can reasonably assume that the program performs as expected. The program performed the MDS on 1024 data set in nearly 11 minuites and the 4096 data set in 2 hours and 40 minuites.

Figure 1. Resulting clusters of Multi Dimensional Scaling. Left hand side images shows the clusters visualized using MeshView software and the right hand side images illustrate the predefined structure expected in the data set using lines drawn on the same image.

Memory Size Limitation

The current MapReduce implementation of the MDS algorithm handles only the matrix multiplication in the MapReduce style. However, I believe that we can come up with a different algorithm which will allow most of the computations performed in MDS to be performed using MapReduce. This will reduce the amount of data that needs to be held in-memory by the master program allowing the MDS program to handle larger problems utilizing the total memory available in the distributed setting.

Hadoop as a Batch Job using PBS

2008-08-20T17:53:00.000-07:00

During my previous data analyses using Hadoop and CGL-MapReduce I had to use the compute resources accessible via a job queue. For this purpose I used the Quarry cluster @ Indiana University which support batch job submissions via Portable Batch System(PBS). I contacted one of the system administrators of the Quarry (George Wm Turner) and he point me to the Hadoop On Demand (HOD) project of Apache which mainly try to solve the problem that I am facing.

We gave HOD a serious try but could not get it working the way we wanted. What we tried was to install HOD in set of nodes and let users to use it on demand via a job queue. This option simply did not work for multiple users, since the configurations options for the file system overlap between the users causing only one user to use Hadoop at a given time.

With this situation, I decided to give it a try to start Hadoop dynamically using PBS. The task the script should perform is as follows.

1. Identify the master node
2. Identify the slave nodes
3. Update $HADOOP_HOME/conf/masters and $HADOOP_HOME/conf/slaves files
4. UPdate the $HADOOP_HOME/conf/hadoop-site.xml
5. Cleanup any hadoop file system specific directories created in previous runs
6. Format a new Hadoop Distribtued File System (HDFS)
7. Start Hadoop daemons
8. Execute the map-reduce computation
9. Stop Daemons
10. Stop HDFS

Although the list is bit long, most of the tasks are straigt forward to perfom in a shell script.

The first problem I face is finding the IP addresses of the nodes. PBS passes this information via the variable PBS_NODEFILE. However, in a multi-core and multi-processor systems the PBS_NODEFILE contains multiple entries of the same node depending on the numebr of processors we requested in each node. So I have to find the "SET" of IP addresses eliminating the duplicates. Then have to update the configuration files depending on this information. So I decided to use a Java program to do the job (with my shell script knowledge I could not find an easy way to do a "SET" operation)

Once I have this simple Java file to perform the steps 1 -4 the rest is straightforward. Here is my PBS script and I the link after the script will show you the simple Java program which update the configuration files.

----------------------------------------------------------
#!/bin/bash
#PBS -l nodes=5:ppn=8
#PBS -l walltime=01:00:00
#PBS -N hdhep
#PBS -q hod

java -cp ~/hadoop_root/bin HadoopConfig $PBS_NODEFILE ~/hadoop-0.17.0/conf

for line in `cat $PBS_NODEFILE`;do
echo $line
ssh $line rm -rf /tmp/hadoop*
done

var=`head -1 $PBS_NODEFILE`
echo $var

ssh -x $var ~/hadoop-0.17.0/bin/hadoop namenode -format
ssh -x $var ~/hadoop-0.17.0/bin/start-dfs.sh
sleep 60
ssh -x $var ~/hadoop-0.17.0/bin/start-mapred.sh
sleep 60
ssh -x $var ~/hadoop-0.17.0/bin/hadoop jar ~/hadoop-0.17.0/hep.jar hep ~/hadoop_root /N/dc/scratch/jaliya/alldata 40 2
ssh -x $var ~/hadoop-0.17.0/bin/stop-mapred.sh
ssh -x $var ~/hadoop-0.17.0/bin/stop-dfs.sh
-------------------------------------------------------------------
[HadoopConfig.java]

As you can see in the script, I use ssh to log into each node of the cluster and perform the cleaning up of the HDFS directories. Then I use SSH to login to the master node to start the Hadoop daemons.

Next comes the actual execution of the data analysis task, which I have coded in the hep.jar.
After the MapReduce computation is over the rest of the commands simply stop the daemons.

This is method has more flexibility to the user and requires no changes for the batch job scheduling system. It also serves as an easy opti

Save Now

on when the number of MapReduce computations are smaller than the other batch jobs.

However, if the cluster is dedicated to run MapReduce computations, then everybody starting and stopping a HDFS file system does not make sense. Ideally the Hadoop should be started by the adminitrators and then the users should be allowed to simply execute the MapReduce computations on it.

Your comments are welcome!

High Energy Physics Data Analysis Using Hadoop and CGL-MapReduce

2008-08-20T17:51:00.001-07:00

After the previous set of tests wtih parallel Kmeans clusting using CGL-MapReduce, Hadoop and MPI I shift the direction of testing to another set of tests. This time, the test is to process large number (and volume of) High Energy Physics data files and produce a histogram of interesting events. The amount of data that needs to be processed is 1 terabyte.

Converting this data analysis into a MapReduce version is straigt forward. First the data is split into managable chunks and each map task process some of these chunks and produce histograms of interested events. Reduce tasks merge the resulting histograms producing more concentrated histograms. Finally a merge operation combine all the histograms produced by the reduce tasks.

I performed the above test by incresing the data size on a fixed set of computing nodes using both CGL-MapReduce and Hadoop. To see the scalability of the MapReduce approach and the scalability of the two MapReduce implementations, I performed another test by fixing the amount of data to 100GB and varying the number of compute nodes used. Figure 1 and Figure 2 shows my findings.

Figure 1. HEP data analysis, execution time vs. the volume of data (fixed compute resources)

Figure 2. Total time vs. the number of compute nodes (fixed data)

Hadoop and CGL-MapReduce both show similar performance. The amount of data accessed in each analysis is extremely large and hence the performance is limited by the I/O bandwidth of a given node rather than the total processor cores. The overhead induced by the MapReduce implementations has negligible effect on the overall computation.

The results in Figure 2 shows the scalability of the MapReduce technique and the two implementations. It also shows how the performance increase obtained by the parallelism diminshes after a certain number of computation node for this particular data set.

July 2nd Report

2008-07-03T21:35:00.000-07:00

CGL MapReduce, Hadoop and MPI

Last few weeks I was busy implementing CGL Map Reduce a streaming based map reduce implementation that uses a content dissemination network for all its communication. Our main objective behind this implementation is to avoid the overhead imposed by the technique adopted by both Google and Hadoop in their map-reduce implementations, that is communicate data via files.

Instead of communicating the data between the map and reduce tasks via files, we use NaradaBrokering's publish/subscribe messaging for the data transfer. In most of the map-reduce use cases the output of the map task is significantly smaller than the size of the input data. In addition, the use of a file system based data communication mechanism is prohibitively slow for Iterative map-reduce tasks such as clustering algorithms. These observations motivates us in implementing the CGL Map Reduce.

Following two graphs compares the performance of CGL MapReduce with other parallization techniques used for SPMD programs. For this benchmark we use kmeans algorithm to cluster a collection of 2D data points.

First we compare the performance of CGL MapReduce with Hadoop and MPI. We increase the size of the data set from 100000 points to 40 million and measured the total time for the computation under different implementations. MPI program was the fastest of the three while CGL MapReduce shows very close performance for large data sets. However, Hadoop's timing is almost 30 times longer than the CGL MapReduce and the MPI.

Figure 1. Performance of CGL MapReduce vs. Hadoop vs. MPI

Next, we performed the similar computation on a Single multi-core machine to see the effect of various parallelization techniques for this type of computations. Java threads, MPI and CGL MapReduce shows the converging results for large data sizes. In this case the main limitation factor is the memory access and hence the technique with minimum overhead to the to memory wins the battle. In our experiment the MPI performed the fastest and the Java threads performed second while CGL MapReduce is little behind Java threads. Again, Hadoop is about 30 times slower than any of the other programs.
MPI program achieves its performance from its C++ roots. Java threads is faster than CGL MapReduce simply due to the additional overheads in the map reduce implementation. Hadoop's slowness is due to its overhead in creating and retrieving files for the communication.

Figure 2. Performance of CGL MapReduce vs. Hadoop vs. MPI vs. Java Threads

More analyses will follow soon.

May 21st Report - CTS Conference

2008-05-23T09:35:00.000-07:00

Last two weeks, i was preparing for the cts 2008 conference. I had to prepare for my talk and the demonstration. I had to struggle with the new Dell laptop to get the demo working, simply because of the incompatibilities in the software and the necessary drivers. After tweaking some configurations, I was able to install Fedora 7 and ROOT. However, the demo started giving some unpredictable behaviors.

The conference went well, did my talk and did the demonstration as well, the demo crashes few times though.

During the conference, the power went off for the Irvine area and we had to stay without power for almost 16 hours. The organization committee worked hard to get the conference going with some emergency power, and the speakers had to shout the audience since we did not have power for the audio equipments.

After all, it is a very fruitful experience for me. My first talk in a large conference. The keynote speeches and the panel discussions brought a lot of insight into the future of the Collaborative Technologies.

Here are the slides for my presentation.

A ROOT bug when using DLLs

2008-05-13T16:58:00.000-07:00

I have been trying to convert the ROOT & C++ program that extends the NaradaBrokering's publish/subscribe functionalities to ROOT users to Windows. I was able to convert the pure C++ part of it and the DLL works fine in Windows.

Then I tried to use the wrappers for ROOT with the generated dictionary. Code generation and the compilation all worked as expected but when I try the DLL the ROOT crashes printing garbage characters.

I track down the problem for two days and was able to reduce the problem into its minimum form.
It simply boils down to a parameter passing problem related to string values.

I then ask the question from the ROOT Talks and one of the ROOT devs, Bertrand, helped me to track down it further.

Finally, the resolution is a bug in ROOT 5.19.02. From my experiments, I know that it was also present in ROOT 5.18 as well.

Here is the full resolution in the ROOT Talk.
http://root.cern.ch/phpBB2/viewtopic.php?p=27702#27702

According to Bertrand, there will be a new release tomorrow and the bug seemed to be fixed in this release.

Hope it will fix my problem and also no more bugs in my way :))

May 7th Report

2008-05-11T19:29:00.000-07:00

Posters for the CTS Conference

I have created two posters for the CTS 2008 conference. After a week of crash learning Adobe Illustrator I was able to create them in the way I need :)

Here are the two posters.

Hadoop Presentation

2008-04-16T21:07:00.000-07:00

Today I did a small presentation on Apache Hadoop

I went though the documentation they have on HDFS, Map-reduce framework, and the Streaming API. I also had to go through the code to understand some of the functionalities in the framework. So far my idea is that the framework is bit more biased towards the text oriented computations. Probably because the initial computations that they use map-reduce are mainly centered on processing large collection of documents (specifically web pages)

Here is my presentation

April 9th Report

2008-04-11T13:48:00.000-07:00

Platform shift for the NBC++ Bridge (Linux to Windows)

The HEP solution is taking a new turn by moving on to CherryPy and Windows. This impose a new requirement for the NaradaBrokering's C++ bridge I wrote.
Initially I did the development in Linux platform using g++ compiler. With the new requirement, I had to compile this for Windows based platforms.

Dr. Julian Bunn gave a big help by converting the C++ bridge to a DLL for windows. After doing some debugging, I was able to get NBC++ working on Windows. Another dimension for NaradaBrokering users.

Since it is hard to maintain two code repositories (one for Windows and one for Linux), I decided to merge the two source code repositories. After covering the differences with pre-compiler directives I was able to get the same code working in both Windows and Linux.

Have to finish the documentation and then I can release the new version of NBC++ soon.

February 27th Report

2008-02-28T20:25:00.001-08:00

Writing Papers

The paper, A Collaborative Framework for Scientific Data Analysis and Visualization, I submitted to CTS2008 got accepted. However, reviewers pointed out that there are some grammatical errors in my writing. I had to correct these errors within a week because of the final submission deadline. The following section highlights my experience with the above.

Dr. Shrideep Pallickara pointed me to two books.
1. The Elements of Style by William Strunk Jr. and E.B. White.
2. The Chicago Manual of Style by University of Chicago Press Staff

I found both books in the bookstore and noticed that the second one is more suitable as a reference. The first one is a really nice book for my situation. It is very small book (~100 pages) but it has lot of grammatical styles with examples in it.

I was able to fix lot of grammatical errors present in my initial write up with the help of the above book.

Still I was not 100% sure about my corrections, and decided to search for a proof read service. I found many online services, which does proof reading. However, all of them charge very high amounts for quick jobs.

Lucy Buttersbry, the secretary of our department, pointed me to Writing Tutorial Services in Ballantine Hall 206 (855-6738). If you are an IU student, this is a very good service. They will not proof read your papers, but will help you to identify the common errors you made by going through the paper.

I showed up my revised paper to one of the instructors and he showed me few more common errors that I had in my writing.

After all these steps and few more review cycles I was able to come up with the final version of the paper.

Formatting the Paper

I did the initial writing on Microsoft Word and converted it to pdf before submitting. After converting the document to pdf, I noticed that the column width(for two column pages) of the pdf document is smaller than the column width expected by the conference format sheet.

I decided to give it a try with the Latex format sheet that they provide. After copying few pages I noticed that the column width is correct and also the length of the paper is slightly reduced when formatted using Latex. Of course, the neatness is superb as well.

At the beginning I found it bit hard to insert figures, but it is simply a matter of finding the (right) easy method of doing it. It is very easy to insert images as pdf files. So the only change I had to do was to convert the png images I had into pdf files. I could easily do that using the CutePDF writer. The following latex section shows how I include figures.

\begin{figure}[h]
\begin{center}
\includegraphics[width=0.46\textwidth]{D:/Academic/Ph.D/CTS2008/architecture2.pdf}
\textbf{\caption{\centering{Architecture of the Proposed Collaborative Data Analysis Framework}}
\label{fig:arch}}
\end{center}
\end{figure}

One final thought. If you have time, better to format the paper using Latex as it save you space as well as provide a very neat paper.

February 13th Report

2008-02-04T07:28:00.000-08:00

Demonstration to Prof Malcolm Atkinson.

Prof Malcolm Atkinson is the Director of The e-Science Institute and e-Science Envoy National e-Science Centre and was visiting our lab on 4th of February. The lab has organized a demo session so that most students can show their work to him.

When I got the mail regarding the demo, I was in the middle of modifying the ROOT-NB-Clarens application(my research prototype) to add new features. After two long days I was able to get it working and today we did the demo successfully.

January 30th Report

2008-02-04T07:16:00.000-08:00

Two papers were due during these and had to struggle with time to get those two papers completed before the deadline.
Followings are the conferences:

Dr. Shrideep and Prof. Fox helped me a lot in refining my ideas and correcting presentation errors. Finally I was able to submit the two papers before the deadline.

I also created a research page which highlights the motivation , goals and the proposed solution of my research. I looking forward to maintain that web site throughout my Ph.D. research.

January 16th Report

2008-01-16T08:57:00.000-08:00

Root Client Supports Shared Eventing and Shared Display Type Collaborations.

So far the HEP Data Analysis Client that we have uses a shared event model for collaboration. All the clients perform the fitting and merging of histograms received from servers. This is a very useful feature if different collaborative clients need to "fit" different models to the data received to them. However, if the same model is used by all the clients, then a shared display type collaboration would be the right solution.

I added a feature to the client so that it publishes its current histogram as an image (after fitting and merging) to a topic using the NB's C++ client. I also developed a separate program to simply subscribe to a topic and display the images received over the pub/sub communication channel. This program is very lightweight as its task is merely showing images in a canvas as an when they are received.

This implementation enables the shared display type collaboration among the participating clients to an experiment. Physicists who just need to see the results of an experiment can simply use the shared display client.

Currently all the clients subscribed to a particular topic will receive the histogram images. However, with the introduction of the "agents" , which keep track of on going experiments, to the system these settings can be controlled.

December 26th Report

2007-12-31T12:58:00.000-08:00

Scalability of the Rootlet Architecture:

Last two weeks I was working on improving the HEP(High Energy Physics) data processing implementation, so that I can do a benchmark on the scalability of the proposed architecture. As the first step, I was able to benchmark the Naradabrokering's C++ Client that I wrote. The following graph compares the performance of Naradaborkering's Java Client vs. C++ Client.

The graph measures the time for two hops (in milliseconds) for various message sizes. The reason for the step wise increase that the Java client demonstrates is mainly the buffer allocation strategy in Java sockets. During the benchmark a message rate of approximately 50 messages per seconds was maintained.

Next, I measured the time for two hops for a 100KB message with increasing message rates. The results shows that the both Java and C++ implementations show stable performance upto the measured 1000 messages per second message rate. According to the results, the C++ Client performs better than the Java clients for higher message rates. (Please see the graph below)Next Step:
The next task is to measure the scalability of the HEP data processing implementation as a whole. For this I am trying to process large amount of HEP data by increasing the number of processing nodes to process the same amount of data so that we can gain performance improvements by splitting the computation task among multiple processing entities.

MapReduce:

Prof. Fox pointed me to few interesting papers(listed below) which discuss on a technique to parallelize large data processing tasks, named MapReduce, which has its roots in functional programming. Right now I am reading the papers and was simply amazed by the similarity of the work we have done so far the and technique described by these papers:

J. Dean and S. Ghemawat, “Mapreduce: Simplified data processing
on large clusters,” in OSDI’04: Sixth Symposium on Operating System
Design and Implementation, December 2004.

R. Pike, S. Dorward, R. Griesemer, and S. Quinlan, “Interpreting the
data: Parallel analysis with sawzall,” Scientific Programming Journal
Special Issue on Grids and Worldwide Computing Programming Models
and Infrastructure, vol. 13, no. 4, pp. 227–298, 2005.

M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, “Dryad:
Distributed data-parallel programs from sequential building blocks,” in
European Conference on Computer Systems (EuroSys), March 2007.

H. chih Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker, “Map-reducemerge:
Simplified relational data processing on large clusters,” in Proc.
SIGMOD, 2007.

Hope to discuss them more in my next blog.

Decmeber 12th Report

2007-12-14T12:32:00.000-08:00

TCSC Symposium Proposal:

After SC07 my main target was to write a paper for the above symposium. According to their website;
"The IEEE TCSC Doctoral Symposium provides a forum for students in the area of Scalable Computing to obtain feedback on their dissertation topics and advice on initiating a research career."

I was able to draft a proposal documentation and then with lot of help from Prof. Fox and Dr. Shrideep we were able to submit it before the deadline.

I learnt a lot regarding writing papers especially in presenting ideas. Coming from the programming background, I always tend to go into details straight away. Sherideep helped me to correct this in the paper.

The paper present our plans on designing a "Scalable Framework for Collaborative Analysis of Sceintific Data" especially for data with the "composition" property. That is, the data analysis task can be broken down to set of sub analyses which can be executed concurrently and merge or combine the results of these sub analyses to form the final results.

November 28th Report

2007-11-24T21:17:00.000-08:00

After month of silence:
During the last month I was completely engaged in getting the ROOT client working with Naradabrokering and Clarens so that the Physicist can submit and monitor analysis jobs collaboratively. It was not just another implementation problem, but a implementation+integration task which requires different hetrogenous components to work together. The user case that we tried to achieve.

A Physicist identify a dataset from a partial physics experiment
He then write an analysis script based on some analysis criteria using ROOT language and test it using a sample data file in his computer.
Now he needs to execute this analysis on all the data files available for a particular experiment.
While the jobs are getting processed, he should be able to monitor the results of each analysis sub task, which is a histogram.
The client program that needs to be developed should be able to display and merge the resulting histograms in real time.
Also, any other physicist who would like to see the result of the analysis as an when it is happening, should be able to connect to the same experiment and see the results getting merged one by one in his Client Software.

With a month of work I was able to implement a Client software written in ROOT language that can achieve all of the above requirements. We were able to show this demonstration during the Supercomputing Conference 07 in Reno Nevada. The following image shows the software while executing an analysis on files located in three different servers.

Some explanation about the software:

Main canvas shows the histogram of the results. All the histograms generated at each analysis sub task is merged and displayed to the user
Panel at the top right; shows the connected server. In this example, it has been connected to three Clarens servers running in three different machines.
Panel below that; shows the available data files at each server. This panel also shows the status of each file, whether it has been processed or not by changing color. "Grey" color indicates that the file has not yet been processed, "Red" color indicates that the resulting histogram for that file has been received and in the process of merging it with the available results so far, and the "Blue" color indicates that the file has been processed and the resulting histogram has been merged with the existing results.

Few Implementation Details:
The GUI is completely written in the interpreted language provided by the ROOT framework.
It uses a C++ bridge for Naradabrokering and C++ client library for Clarens server internally.
It also uses a python script to submit analysis tasks to multiple Rootlets (A concept similar to Servlet) concurrently. This is especially required because the interpreted ROOT language does not support multi-threading.

October 3rd Report

2007-10-01T09:02:00.000-07:00

Had a meeting with Prof. Geoffrey relating to the Ph.D. topic that I should select. He advice me to find more use cases of the data analysis tasks that are similar to the Particle Physics data analysis that we are doing using Clarens and ROOT.

The data analysis tasks that we handled has the following characteristic.

Data is in large files and these files are distributed across the globe.
One or more analysis technique(in our case, one analysis script) can be applied to all the data to identify patterns.
The outcome of an analysis of a single file is a histogram.
These outcomes(histograms) can be merged to produce the final results.

So far I have found one strong use case of this nature and that is;
Astronomical Image Processing - mainly for identifying features in astronomical images.

There are few candidate areas that I found interesting and they are;
Analysis of Earthquake Data
Microarray Analysis for Genes
Pattern Matching in Financial Data

Currently I am reading to find out the exact data analysis requirements of these fields. The target is to find more use cases for "Distributed Composable Data Analysis"

September 19th Report

2007-10-01T08:56:00.000-07:00

Conrad tested the demo from CERN and it worked well. So, now I can focus on the next step of the project.
Conrad also sent me a link to more root data files so as the next step I will test the demo with those new root data files. The first demo only uses a single rootlet and the reason for this is mainly the way how the Clarens client is written. Each analysis request is processed synchronously and hence the client send requests one by one to the server for each root data file to be analyzed.

As the next step of the project, I am planning to run the client in with multiple processes and allow it to create multiple rootlets so that the analysis can be performed simultaneously utilizing the full cpu power. Hope to get the results soon.

September 5th Report

2007-09-13T18:08:00.000-07:00

Demo is ready.

ROOT application development supports both interpreted code and also compiled code. Interpreted code (Mainly the analysis code written by physicist) can use compiled shared libraries including ROOT provide ones and also other user written libraries. Interpretation helps the user to debug and fix code easily and write the necessary analysis as a function.

As explained in my previous blog (August 22nd Report) the plan is to let the rootlet to publish the location of the generated histogram files to the subscribed clients. Clients who receive those notifications then use these files to update their results.

As the second phase, I developed a ROOT compliant wrapper classes for the C++ client of Naradabrokering. These allow ROOT clients to utilize the publish/subscribe capabilities of Naradabrokering.
The steps for developing the ROOT compliant classes are explained in the following tutorial.
Part1, part2 and part3

I was able to get the publishing of messages working from interpreted code and thought that the subscription would work in the same manner. After few days of trying I found that I was wrong.
To get the subscription to work, the interpreted code should pass a function pointer as the callback to the compiled code and on the reception of a notification , the compiled code should call this callback (which is in the interpreted code). Passing a function pointer to the compiled code is easy and straight forward, however, when the compiled code try to call that pointer I got an error *****Segmentation Violation*********

This happens mainly because of the limitation of the ROOT interpreter in resolving function pointers across the interpreted/compiled code boundaries. After few days of searching and querying Conrad, we decided to ask the question on the ROOT mailing list. They replied really soon and helped us to solve the problem. The solution came in a way of a reflection type call interface for calling interpreted code from the compiled code supported by ROOT.
Here is the mail thread.
http://root.cern.ch/phpBB2/viewtopic.php?t=5408

So with that help, I was able to get the demonstration working and it is so nice to see how it is working. Results of the remote analysis triggers users who are subscribed for notifications and their histograms get updated with remote results.

Somethings are too good to be true!

August 22nd Report

2007-08-22T11:26:00.000-07:00

Last week Conrad helped us in setting up a Clarens server in gridfarm003 and after resolving few issues with our certificates I was able to use it.

Usage Scenario : Big Picture

The user writes a Client code in C++ which utilizes services of the clarens server. Let's call this ClientCode.C
She also has written the analysis code for root data. Let's call these files Analysis.C and Analysis.H .
She then executes the ClientCode.C using the C++ interpreter provided by the ROOT. (ROOT has a built in C++ interpreter)
ClientCode.C uses the built in ROOT libraries to locate data files in the server and also to upload the above two files to the clarens server.
After discovering (polling for files) root data files, ClientCode.C send request/requests to the Clarens server for creating rootlet/rootlets to execute the analysis code that it uploaded.
For every rootlet request Clarens server creates a wrapper script for rootlet and executes it with the input and output files. This wrapper script is called rootlet_wrappper.sh
Finally the ClientCode.C poll for output files and display the results of the analysis in a histogram generated at the user's machine.

Incorporating Naradabrokeing

As the first step, I changed the rootlet_wrapper.sh to publish a message after finishing the analysis using the nbclient program we wrote using C++. This works fine and we can eliminate the polling requirement of the ClientCode.C to find the results.

Next task is to incorporate the subscriber functionality to the visualization part of the ClientCode.C

August 8th Report

2007-07-31T07:38:00.000-07:00

Secure Message Transfer Between Java and C++

Scenario:

We are developing an application which require secure message transfer between Naradabrokering (java based messaging substrate) and C++ client application. The communications between the entities uses a custom publish/subscribe messaging protocol to get better performance. (No XML processing)

JDK has a built in support for security features such as certificate handling, encryption and signing. However, to get those functionalities in C++ a separate library needs to be installed. For this we used Openssl (http://www.openssl.org/) To develop applications it is required to have the development files of the openssl and the installation is different according to the underlying operating system. For my machine running Ubuntu 2.6.15-28-386 it is simply;

apt-get install libssl0.9.7
apt-get install libssl-dev

Following sections of shows the code fragments that we can use to encrypt/decrypt messages (bytes) both in Java and C++. The algorithm used for the encryption is AES (http://en.wikipedia.org/wiki/Advanced_Encryption_Standard)

Encryption in JAVA

In java the encryption is handled by the provided javax.crypto.Cipher class. The following code fragment shows the encryption in Java.

byte[] bytesToEncrypt = /*Bytes to be encrypted*/
byte[] encBytes = null; /*Encrypted Bytes*/

/**
* Create a Cipher by specifying the following parameters a. Algorithm
* name - here it is AES */

Cipher aesCipher;
try {
aesCipher = Cipher.getInstance(Constants.AES_ALGO);

aesCipher.init(Cipher.ENCRYPT_MODE, secretKey);
encBytes = aesCipher.doFinal(msg.getBytes());
} catch (Exception e) {
throw new ClarensException(
"Error encrypt message using secret key",e);
}

These bytes are then transferred to the C++ client using socket based communication channel.

Decryption in C++

Openssl provides a set of libraries for handling the decryption and the following utility function shows how we can use those to decrypte the received set of bytes.

bool
SecurityUtil::decryptAES(const unsigned char *in,int inputLength ,unsigned char *out,int &outputLength, string aesKey){

int olen, tlen, n;
EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init (& ctx);
EVP_DecryptInit (& ctx, EVP_aes_128_ecb (), (unsigned char *)aesKey.c_str(), NULL);

olen=0; tlen=0;

if (EVP_DecryptUpdate (& ctx, out, & olen, (const unsigned char*)in,inputLength) != 1)
{
cerr<<"error in decrypt update"< return false;
}

if (EVP_DecryptFinal(& ctx, out + olen, & tlen) != 1)
{
cerr<<"error in decrypt final"< return false;
}

olen += tlen;
outputLength=olen;

EVP_CIPHER_CTX_cleanup (& ctx);
return true;
}

Ok, now let's see the other side of the story, from C++ to Java

Encryption in C++

Again Openssl provides a set of library functions for encryption as well. Following is the utility function for the encryption

bool

SecurityUtil::encryptAES(const unsigned char* in,int
inputLength ,unsigned char *out,int &outputLength, string
aesKey){

int olen, tlen, n;
EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init (& ctx);

EVP_EncryptInit (& ctx, EVP_aes_128_ecb (), (unsigned char
*)aesKey.c_str(), NULL);

if (EVP_EncryptUpdate (& ctx, out, & olen, (const unsigned
char*)in , inputLength) != 1)
{
cerr<<"error in decrypt update"<

if (EVP_EncryptFinal (& ctx, out + olen, &amp;amp;amp;amp;amp; tlen) != 1)
{
cerr<<"error in encrypt final"<

olen+=tlen;
outputLength=olen;

EVP_CIPHER_CTX_cleanup (& ctx);
return true;
}

Decrypting the bytes received from the C++ client in JAVA

Decryption is java is also handled by the javax.crypto.Cipher class and is fairly straight forward. Here is the code segment.

byte[] decBytes = null;

Cipher aesCipher;

try {
aesCipher = Cipher.getInstance(Constants.RSA_ALGO);

aesCipher.init(Cipher.DECRYPT_MODE, prKey);
decBytes = aesCipher.doFinal(msgBytes);
} catch (Exception e) {
throw new ClarensException(
"Error decrypting message using private key ", e);
}

Simple right? The main problem I faced when developing the above application was the lack of documentation on this regard. There are tons of documentation on how to handle encryption/decryption using java but very small number for the same in C++. How about encryption/decryption between Java and C++? I could not find anything in this sort. Openssl has a good documentation on various functions/data structures it offers for encryption/decryption but the main problem is there very limited amount of code examples which shows the exact usage. Followings are some of the resources that I used to come up with this implementation and hope someone will find this helpful.

http://www.openssl.org/docs/
http://www.madboa.com/geek/openssl/#cert-self
http://www.ibm.com/developerworks/linux/library/l-openssl.html
http://www.mail-archive.com/openssl-users@openssl.org/msg40449.html
http://www.mail-archive.com/openssl-users@openssl.org/msg23119.html
http://www.fortrel.net/blog/index.php?title=encryption_java_c&more=1&c=1&tb=1&pb=1
http://www.adp-gmbh.ch/cpp/common/base64.html

Next Blog: Signing and Verifying between Java and C++

July 25th Report

2007-07-31T07:16:00.000-07:00

During the last two weeks i was mainly focusing on two things.
1. Qualifying Exam (on 19th of July)
2. Finishing the Service Discovery Framework for Clarens

Results:
I passed the Oral Qualifiers. However, from the questions asked from me, I realized that there are lot more that I should learn.

Service Discovery Framework has some more work. Getting the encryption (both using symmetric keys and RSA keys) working with C++ and Java seems bit tough. Main reason for this is the lack of documentation.

July 11th Report

2007-07-11T15:15:00.000-07:00

Last week I was able to complete the functionality of the agent discovery for the C++ client. After that I put the client into a load test and found that it crashes after few (correct)discovery cycles.

Since it works correctly for at least one discovery the error is not in the logic but in the implementation details. It is implemented in C++ and the debugging is not that easy when there are some threads in the code. After a lot of (a lot of actually) debugging I found the problem.
It was just an update of boolean variable that I missed. (my eyes could not catch it during the first few debug cycles.)

After that I went through the code again, and did a complete cleanup. Then I let the test run for few hours and found that it is running without any problems and most importantly without any memory leaks.

June 27th Report

2007-06-21T08:19:00.000-07:00

Axis2 Hackathon June 11th to June 16th.

We started the hackathon with the intention of cleaning up the bugs list and come with some API fixes. It was a successful hackathon and we (Srinath Perera, Eran Chinthaka, Deepal Jayasinghe, Amila Suriyaarchhi, Ajith Ranabahu, Glen Daniels, and Myself) were able to fix most of the bugs whcih were labled as blockers and do some cleanup in the API as well. Axis2 has now grown into a large project and the API changes are not very straight forward as we did one year back. It is now a process of improvement plus depreciation.

An excellent presentation on API Design.
http://www.infoq.com/presentations/effective-api-design

C++ Agent/Service Discovery for Clarens

I was able to get the C++ client working for the Agent Discovery. The *next step* is to get the service discovery working which require both encryption and signing of messages.

Getting the exchange of messages between java and C++ was not easy with security in place. I am using openssl libraries for the C++ client and for the java side, it is the built in cryptographic extension of JDK. However, there are compatibility issues when it comes to different versions of certificates and padding schemes. So I will write a blog with complete code samples regarding the above issues after getting that *next step* completed.

June 13th Report

2007-06-21T08:12:00.000-07:00

Started working on the C++ version of the Clarens implementation. I need to implement the client code of the proposed architecture in C++ so that both C++ and Python clients can use it.

The initial plan was to develop a version without security, but it seems like Caltech needs security in all their applications. It will not be easy to inject security related code once developed without it, so I decided to implement the complete version of the client code with security similar to the java implementation.

Currently I am implementing the above code and it is a bit of slow going process because I have to learn how to use OpenSSL libraries for security functions.

In the meanwhile I was able to schedule my Oral exam with my professors and the data I could get was 19th of July. Currently preparing for the exam as well.

May 30th Report

2007-06-21T08:09:00.000-07:00

Continued working on the Clarens Project :

I created a python service "ping.ping" which responds to an agent's requests informing that the service is up and running and deployed it in clarens.

And also tested the following scenario;

First, a RootletSimulator(java) registers the rootlet service with an agent. (In the actual implementation this should be handled by the initialization method of the rootlet service)

After the registration, the agent will keep on pinging the rootlet service (python running on Clarens) to make sure that the service is up.

Now, if we use the ClientSimulator (java) it can discover the service and may be use the service.

Then we can stop the Clarens server or remove the rootlet service from the web-frontend (need to find out how to do this)

Now if we use the ClientSimulator(java) again, it won't discover the service, because the agent has removed the service from its list of available services.

Security Considerations:

Agent discovery request/response uses signed messages.
Service discovery request/response uses encrypted and signed messages.
*pings* use xmlrpc over http and the messages are secured using a grid-proxy.

May 15th Report

2007-05-15T21:41:00.000-07:00

Started working on the C++ Bridge for Nardabrokering.
Current work includes, adding a layer of functions to the bridge so that it can be used to discover agents and rootlet services in architecture comprises of multiple agents and rootlet services.

Started developing the C++ client and then started implementing the agent (java) as it is required to test the C++ client code.

To provide secure agent/rootlet discovery mechanism architecture was discussed that utilize both symmetric and asymmetrical keys. The proposed message exchange pattern for discovering an agent and rootlet service is as follows.

Client ------------------------------------------- Agent

->DiscoverAgentRequest with Client’s credentials.

<- DiscoverAgentResponse encrypted using a secret key
Secret key is encrypted using the Client’s public key
Message is signed using Agent’s Private key
Message contains agents credentials.

->DiscoverRootletRequest with Client’s credentials
Message encrypted using the secret key (Shared during the previous step)
Signed using Client’s Private key

<- DiscoverRootletResponse Message encrypted using the secret key
Signed using the Agent’s Private key

Since we cannot exchange objects in their native serialization form from java and C++ a customer serialization mechanism is designed to transfer data between the clients and agents.

Currently working on getting the security framework working in both java and C++.

My Version of the Spring Semester

2007-05-04T12:27:00.000-07:00

January - Diarrhea + Vomiting
- My wife got Chickenpox
February - Chickenpox
- Reactive Arthritis due to chickenpox
- Enlargement in my thyroid gland
March - Went back home
April - Thyroid Surgery
May - Came back to US

January 24th Report

2007-01-22T10:47:00.000-08:00

During the last two weeks I completed the required benchmarks for the paper "A Scalable Approach for the Secure and Authorized Tracking of the Availability of Entities in Distributed Systems, Shrideep Pallickara, Jaliya Ekanayake and Geoffrey Fox, which was accepted for the IPDPS 2007 .

In addition I was researching on the Hessain library which can be used in the data transformation layer of the C++ bridge for Naradabrokering.

December 13th Report

2006-12-13T11:15:00.000-08:00

Finished implementing the C++ client for the Naradabrokeing project.
Few more pending tasks to call it really "Finished"

-Check the portability between different operating systems
-Check for different platforms (currently it runs on a 32 bit Intel based Linux box)
-Benchmarks

NOVEMBER 29th Status Report

2006-11-29T07:55:00.000-08:00

During the last two weeks I have been implementing the C++ client for the Naradabrokering.

First I tried connecting to the broker and establish a connection. It worked and I found that the Java uses BigEndian scheme when they serialize data types where as Intel based architectures use LittleEndian scheme. So when serializing data types like int, long from C++ need to re-arrange the bytes.

Now I am converting the required classes and try to get the message publishing to work from C++.

Converting an existing java application to a C++ application is always not easy as there are many simple utilities that are available in Java and missing in C++.

One of them is the string operations (C++ has string operations, but since we are working with event headers we have to use char operations.

Java code always has the luxury of using different collection classes to keep object references but when it comes to C++ they need to be implemented manfully.

NOVEMBER 15th Status Report

2006-11-13T13:21:00.000-08:00

NOVEMBER 15th Status Report

Status Report for 10/18/2006 to 01/11/2006

2006-11-01T06:55:00.000-08:00

Implemented a C++ Bridge for Naradabrokering. The project was completed as per the initial requirements and waiting for the feedback.

Started working on the initial steps of my research.