tag:blogger.com,1999:blog-369489282024-03-07T00:04:47.073-08:00Open Notebook[my notebook in the cloud]Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.comBlogger55125tag:blogger.com,1999:blog-36948928.post-5597097386470401972014-02-14T11:03:00.003-08:002014-02-14T11:03:50.903-08:00Offline Disk - Make it Online.I noticed that one of the disks in my machine is set to offline and the Disk Management tool was not giving an option to make it online. Thanks to following blog, I was able to fix the issue.<br />
<br />
<em><a href="http://www.happysysadm.com/2010/11/disk-is-offline-because-of-policy-set.html">the disk is offline because of policy set by an administrator</a></em><br />
Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-4624547326288011652010-07-16T10:27:00.000-07:002010-07-16T10:28:33.751-07:00Azure Platform Appliance<p style="LINE-HEIGHT: normal; MARGIN: 0in 0in 0pt" class="MsoNormal"><span style="FONT-FAMILY: 'Georgia', 'serif'; FONT-SIZE: 12pt; mso-fareast-font-family: 'Times New Roman'; mso-bidi-font-family: 'Times New Roman'">Microsoft recently announced its next move with Azure "<a href="http://www.microsoft.com/windowsazure/appliance/">Azure Platform Appliance</a>". The idea is to offer Azure platform as an infrastructure that can be used in private clusters. From my perspective, this is somewhat similar to the idea of private clouds (infrastructure services) that one can deploy using runtimes such as Eucalyptus or Nimbus on local clusters. However, since Azure is not an infrastructure service, it will be more flexible for the users. Unlike pure virtual machines, Azure platform appliance will expose most platform services of Azure as well. In addition, the migration between the private Azure and the public Azure will be seamless as well. Overall, I think the biggset advantage of this approach is the piece of mind that the businesses will have "I am in control of my data and it is local".<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p></span></p>Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com6tag:blogger.com,1999:blog-36948928.post-37150363926991019102010-06-24T16:12:00.000-07:002010-06-24T16:13:51.135-07:00Azure Storage Browser in Visual StudioNow <a href="http://blogs.msdn.com/b/jnak/archive/2010/06/10/windows-azure-storage-browser-in-the-visual-studio-server-explorer.aspx">this </a>is a handy tool.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com1tag:blogger.com,1999:blog-36948928.post-86321754837381376012010-06-11T14:54:00.001-07:002010-06-11T14:54:21.564-07:00Hadoop Application WikiA nice collection of Hadoop apps.<br /><a href="http://wiki.apache.org/hadoop/PoweredBy">http://wiki.apache.org/hadoop/PoweredBy</a>Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com2tag:blogger.com,1999:blog-36948928.post-63645008728360824282010-03-18T21:31:00.000-07:002010-03-18T21:37:51.336-07:00What is the concept of "static data" in Twister?Many iterative applications we analyzed show a common characteristic of operating on two types of data products called static and variable data. Static data (most of the time the largest of the two) is used in each iteration and remain fixed throughout the computation whereas the variable data is the computed results in each iteration and typically consumed in the next iteration in many expectation maximization (EM) type algorithms.<br /><br />If a set of data is read by the application but do not get changed, then this set of data can be considered "static" in Twister. Matrix blocks, points in clustering, and web graph in pagerank are all examples of static data.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com1tag:blogger.com,1999:blog-36948928.post-3478552624761410662010-03-18T21:27:00.000-07:002010-03-18T21:38:26.467-07:00How does Twister differ from MRnet?There are many differences between <a href="http://www.paradyn.org/mrnet/">MRnet </a>and MapReduce and then with Twister. I will list some here.<br /><br />In MapReduce the communication between map and reduce stage of the computation is a graph not a tree. A map can produce (key,value) pairs that may end up in multiple reducers. In MapReduce the framework does not impose any communication topology or connection topology between map and reduce stages of the computation. It is purely the intermediate keys that determine the communication pattern. For example, in an associative and commutative operations such as sum or histogramming, how the intermediate keys are used to distribute intermediate results among the reduce tasks is not that important.However for operations such as sorting, or matrix operations, one can select the intermediate keys in such a way that specific keys goes to specific reduce tasks. Again this is not defined by the network, but the keys and the key selector functions.<br /><br />Twister uses pub/sub messaging to implement a MapReduce runtime,especially to support iterative MapReduce computations. Similar to other MapReduce runtimes it gives more focus on processing data while maintain data- process affinity. Map and reduce functions in Twister are long running processes providing distinction between static data and variable data. It supports broadcast, scatter type data distributions and reading data via the local disks. I am not sure how the latter two functions can be handled using MRnet.<br /><br />Comparing MRnet with MapReduce for data processing applications is an interesting analysis one can do though.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com2tag:blogger.com,1999:blog-36948928.post-47120807896843133182010-02-09T15:26:00.000-08:002010-02-09T19:03:08.398-08:00Twister: Iterative MapReduceMapReduce programming model has simplified the implementations of many data parallel applications. The simplicity of the programming model and the quality of services provided by many implementations of MapReduce attract a lot of enthusiasm among parallel computing communities. From the years of experience in applying MapReduce programming model to various scientific applications we identified a set of extensions to the programming model and improvements to its architecture which will expand the applicability of MapReduce to more classes of applications.<br /><br />Twister is a lightweight MapReduce runtime we have developed by incorporating these enhancements. We have published several scientific papers [1-5] explaining the key concepts and comparing it with other MapReduce implementations such as Hadoop and DryadLINQ. Today we would like to announce its first release.<br /><br />Key Features of Twister are:<br /><ul><li>Distinction on static and variable data</li><li> Configurable long running (cacheable) map/reduce tasks</li><li> Pub/sub messaging based communication/data transfers</li><li> Combine phase to collect all reduce outputs</li><li> Efficient support for Iterative MapReduce computations (extremely faster than Hadoop or DryadLINQ)</li><li> Data access via local disks</li><li> Lightweight (5600 lines of code)</li><li> Tools to manage data </li></ul>For more details please visit <a href="http://www.blogger.com/www.iterativemapreduce.org">www.iterativemapreduce.org</a> and let us know your thoughts and experience using Twister.<br /><br />Thank you,<br /><br /><a href="http://salsaweb.ads.iu.edu/salsa/">SALSAHPC Team</a>.<br /><br />[1]. Jaliya Ekanayake, (Advisor: Geoffrey Fox) <a href="http://grids.ucs.indiana.edu/ptliupages/publications/SC09-abstract-jaliya-ekanayake.pdf">Architecture and Performance of Runtime Environments for Data Intensive Scalable Computing</a>, Doctoral Showcase, SuperComputing2009.<br />[2]. Jaliya Ekanayake, Atilla Soner Balkir, Thilina Gunarathne, Geoffrey Fox, Christophe Poulain, Nelson Araujo, Roger Barga, <a href="http://grids.ucs.indiana.edu/ptliupages/publications/eScience09-camera-ready-submission.pdf">DryadLINQ for Scientific Analyses</a>, Fifth IEEE International Conference on e-Science (eScience2009), Oxford, UK.<br />[3]. Jaliya Ekanayake, Xiaohong Qiu, Thilina Gunarathne, Scott Beason, Geoffrey <a href="http://grids.ucs.indiana.edu/ptliupages/publications/cloud_handbook_final-with-diagrams.pdf">Fox High Performance Parallel Computing with Clouds and Cloud Technologies</a> Technical Report August 25 2009 to appear as Book Chapter.<br />[4]. Geoffrey Fox, Seung-Hee Bae, Jaliya Ekanayake, Xiaohong Qiu, and Huapeng Yuan, Parallel <a href="http://grids.ucs.indiana.edu/ptliupages/publications/CetraroWriteupJune11-09.pdf">Data Mining from Multicore to Cloudy Grids</a>, High Performance Computing and Grids workshop, 2008. – An extended version of this paper goes to a book chapter.<br />[5]. Jaliya Ekanayake, Shrideep Pallickara, Geoffrey Fox, <a href="http://grids.ucs.indiana.edu/ptliupages/publications/ekanayake-MapReduce.pdf">MapReduce for Data Intensive Scientific Analyses,</a> Fourth IEEE International Conference on eScience, 2008, pp.277-284.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com4tag:blogger.com,1999:blog-36948928.post-68326147232694330712010-01-03T21:42:00.000-08:002010-01-03T22:12:02.420-08:00Almost MapReduce in 1992<span style="font-size:130%;">Paralex: An Environment for Parallel Programming in Distributed Systems (1992)</span><br /><br /><span style="font-style: italic;">by Özalp Babaoglu , Lorenzo Alvisi , Alessandro Amoreso , Renzo Davoli , Davoli Luigi , Luigi Alberto Giachini </span><br /><br />Just found this paper and read it to the end since I noticed some similarities of what they have proposed in 1992 and the current MapReduce programming model and some of the observations are still true for today as well. I will list few of the observations/assumptions they have made showing the similarity of their work and the current MapReduce programming model.<br /><br /><ul><li><span style="font-style: italic;">Large-grain data flow model suitable for high-latency low bandwidth networks</span></li><li><span style="font-style: italic;">Only by keeping the communication-computation ratio to reasonable levels can we expect reasonable performance from parallel applications in such system.</span> – We noticed a similar thing with performing parallel computing in Cloud infrastructures [<a href="http://grids.ucs.indiana.edu/ptliupages/publications/CGLCloudReview.pdf">paper</a>]</li><li><span style="font-style: italic;">Paralex functions mush be “pure” – no side effects</span></li><li><span style="font-style: italic;">Node corresponds to computations (functions, procedures, programs) and links indicate flow of typed data</span> – Compare this with Microsoft Dryad’s DAG based programming model.</li></ul><a href="http://research.microsoft.com/en-us/projects/dryad/eurosys07.pdf">Dryad paper</a> has referred to their work. - I haven't noticed it before ;)<br /><br />Some of their performance measures had issues with 16MB data set because the memory they had in one of the machines was only 16 MB. Today we have the luxury of using large memories but our data sets are also grown into petabytes. What they did with NFS is now done in HDFS in Hadoop.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com2tag:blogger.com,1999:blog-36948928.post-30658381542422247222009-12-01T12:15:00.001-08:002009-12-01T12:56:01.687-08:00Dynamic Provisioning of Virtual ClustersHere I will present the details of the demonstration that we (<a href="http://www.infomall.org/salsa/">SALSA </a>team) presented at the Super Computing 09 conference in Portland.<br /><br />Deploying virtual/bare-system clusters on demand is an emerging requirement in many HPC centers. The tools such as <a href="http://xcat.sourceforge.net/">xCAT </a>and <a href="http://www.clusterresources.com/products/moab-cluster-suite.php">MOAB </a>can be used to provide these capabilities on top of a physical hardware infrastructure.<br /><br />In this demonstration we coupled the idea of provisioning clusters with parallel runtimes. In other words, we showed that people can switch a given set of hardware nodes into different operating systems (either virtual or non-virtual) and run different applications written using various cloud technologies. Specifically we showed that it is possible to provision clusters with Hadoop on both Linux bare-system and virtual machines and Microsoft DryadLINQ on Windows Server 2008.<br /><br />The following diagram shows the operating system and the software stacks we used in our demonstrations. We were able to get all configurations demonstrated except for Windows HPC on XEN VMs which does not work well due to the non-existence of para-virtualized drivers.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8Xzz3TtLcbc4macezz2_BQXFhbi349adnzpQqMX2NQRIoWzQfw0w1DhwjIWqaQ0CbUUC5SylPYVnFZbNpAEyIayRcibQLX_EeiIEme_ZsGBywknYoqQISR7GynDA0OZEsiXXQpw/s1600/architecture.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 167px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8Xzz3TtLcbc4macezz2_BQXFhbi349adnzpQqMX2NQRIoWzQfw0w1DhwjIWqaQ0CbUUC5SylPYVnFZbNpAEyIayRcibQLX_EeiIEme_ZsGBywknYoqQISR7GynDA0OZEsiXXQpw/s320/architecture.png" alt="" id="BLOGGER_PHOTO_ID_5410372502086848850" border="0" /></a><br />We setup our demonstration in 4 clusters each with 64 CPU cores (8 nodes with 8 CPU cores each). The first 3 clusters were configured with RedHat Linux bare-system, RedHat Linux on XEN, and Windows HPC 2008 operating systems respectively. The last cluster is configured to dynamically switch between any of the above configurations. In Linux (both bare-system and XEN) clusters we ran a Smith Waterman dissimilarity calculation using Hadoop as our demo application while on Windows we ran a DryadLINQ implementation of the same application.<br /><br />We developed a performance monitoring infrastructure based on pub-sub messaging to collect and summarize CUP and memory utilization of individual clusters and a performance visualization GUI (Thanks Saliya for the nice GUI) as our front end demonstration component. Following two diagrams show the monitoring architecture and the GUI of our demo.<br />With the 3 static clusters we were able to demonstrate and compare the performance of Hadoop (both on Linux bare-system and XEN) and DryadLINQ to the people. It also provides a way to show the overhead of virtualization as well. The dynamic cluster demonstrated the applicability of the dynamically provisionable virtual/physical clusters with parallel runtimes for scientific research.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBcVahc9jrHsC3b1NaSfDUPlWn6eTVDeJnMcfnG4LvDVyYEvXSKkrMAvrxtKyYDjzk_9czBP25l0jHBPma-IFA6ZrUWxLNdmat4Nq8ZGycisvlt0DdpK26iMt5GCk1YD9JX73k5Q/s1600/monitoring_architecture.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 181px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBcVahc9jrHsC3b1NaSfDUPlWn6eTVDeJnMcfnG4LvDVyYEvXSKkrMAvrxtKyYDjzk_9czBP25l0jHBPma-IFA6ZrUWxLNdmat4Nq8ZGycisvlt0DdpK26iMt5GCk1YD9JX73k5Q/s320/monitoring_architecture.png" alt="" id="BLOGGER_PHOTO_ID_5410372739699764562" border="0" /></a><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUJ6wbuZjOc-bXab1SroECz_yN75EA45aDSoMDwQHDKp1qswOTcd3bR3AWSe8F8k2yF-aeRN9ni36hMeRpwYTCFqVvBWELJcO7_Cps7ARyiAiPAL5CPqsEEz4VPkDLF3Po2DkGMw/s1600/monitor_full.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 193px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUJ6wbuZjOc-bXab1SroECz_yN75EA45aDSoMDwQHDKp1qswOTcd3bR3AWSe8F8k2yF-aeRN9ni36hMeRpwYTCFqVvBWELJcO7_Cps7ARyiAiPAL5CPqsEEz4VPkDLF3Po2DkGMw/s320/monitor_full.png" alt="" id="BLOGGER_PHOTO_ID_5410372825398866018" border="0" /></a><br />It was a wonderful team work involving all the members of the SALSA team and the members of IU UITS. What is more amazing is that we go this very successful demonstration built from the scratch in less than one month.<br />We will upload a video of the demonstration soon.<br /><br />Here is a photo of most of the members of our group.<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFVCKbCDEjKMyEMxq6ceVn7cguAxBVw741KK46JKHMJYArN6MVXcz0NI6efK1EdhtnYfTrna-mbLEKBHACDhVaYGdeSV50N0i3NOG9bhy9ypkpUh70OUPkqoM1xnSKN2Ef18fTvA/s1600/100_8343.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 240px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFVCKbCDEjKMyEMxq6ceVn7cguAxBVw741KK46JKHMJYArN6MVXcz0NI6efK1EdhtnYfTrna-mbLEKBHACDhVaYGdeSV50N0i3NOG9bhy9ypkpUh70OUPkqoM1xnSKN2Ef18fTvA/s320/100_8343.JPG" alt="" id="BLOGGER_PHOTO_ID_5410373267629665682" border="0" /></a>Front row (left to right): Jaliya Ekanakaye, Dr. Judy Qiu, Thilina Gunarathne, Scott Beason, Jong Choi, Saliya Ekanayake, Li Hui.<br />Second row (left to right) Prof. Geoffrey Fox, Joe Rinkovsky and Jenett Tillotson.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-40064898562196391822009-11-01T21:03:00.000-08:002009-11-01T21:05:16.509-08:00CloudComp09 PresentationI shared the slides presented at CloudComp09 via <a href="http://www.slideshare.net/jaliyae/high-performance-parallel-computing-with-clouds-and-cloud-technologies">slideshare</a>.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-43484581430084265532009-11-01T20:41:00.000-08:002009-11-01T21:01:11.186-08:00Windows Server 2008 on XenAs part of our ongoing research, we started running DryadLINQ applications on a virtual cluster running Windows Server 2008 VMs on top of Xen. Although we noticed acceptable performance degradation with para-virtualized Linux guests (from our previous research), with Windows we noticed extremely higher performance degradations. Also we noticed that whenever we run a DryadLINQ job which has considerable amount of communication, the VM that runs the DryadLINQ job manager crashes. We had an exactly similar experience with Windows VMs provided by GoGrid earlier. However, those VMs had only 1 CPU core where as the VMs we just ran have 8 CPU cores. (Note: This effect has nothing to do with DryadLINQ, but the handling of I/O operations by the guest and the host OSs)<br /><br />Overall, the full virtualization approach may not seem to work well for parallel applications. Especially for applications with considerable inter-process communication requirements. Following is a brief analysis I wrote.<br /><br />===========================<br />If we look at the development of the virtualization technologies, we can see these three virtualization techniques.<br /><br />Full virtualization -> Para-Virtualization -> Hardware Assisted Virtualization<br /><br />(e.g. VM Ware Server) e.g. Xen, Hyper-V e.g. Hyper-V, Virtual Iron, VM-Ware Workstation (64 bit)<br /><br />So far, para-virtualization seems to be the best approach for virtualization to achieve better performance. However, it requires modified guest operating systems. This is the problem for Windows guests, and that is why we cannot run Windows Server 2008 on Xen as para-vritualized guest. According to [1] the hardware assisted approach is still in its early stages and not performing well, but I think this will catch up soon.<br /><br />Hyper-V coming from Microsoft, may provide better virtualization solutions to Windows guests and currently it supports both para-virtualization and hardware assisted virtualization.<br /><br />Given these observations and the observations from our tests, currently it does not seem feasible to have one virtualization layer (like Xen) on a cluster and provide both Windows and Linux VMs for parallel computations. (Here I assumed that Linux on Xen is performing better than Linux on Hyper-V, we need to verify this).<br /><br />However, if we go for a hybrid approach such as using technologies like XCAT to boot Windows and Linux bare-metal host environments, and provide Linux virtualization using Xen and Windows virtualization using Hyper-V, we may be able to utilize best technologies from both worlds.<br /><br />Then the next thing to figure out is the performance and how we can run Dryad, Hadoop, and MPI on these virtual environments.<br /><br />Although we did not see this “<span style="font-weight: bold;">mythical” 7</span>% virtualization overhead, they are not so bad as well (on private clouds /clusters at least – we saw 15% to 40% performance degradations).<br /><br />However, we need to figure out ways of handling large data sets in VMs to use Hadoop and Dryad for wide variety of problems. The main motivation of MapReduce is “moving computation to data”. In virtualized environments, we currently store only an image of the VM without data. If the data is large we have to either move them to the local disks of virtual resources before we run computations, or attach blobs/block storage devices to VMs. The first approach does not preserve anything once the virtual resource is rebooted, whereas the second approach adds network links in between the data and computations (moving data). If our problems are not so data intensive, we can simply ignore these aspects for the moment, but I think this is something worth investigating.<br />==========================<br />Hope to add more results soon.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-82760344903406812232009-09-17T08:37:00.001-07:002009-09-17T08:38:38.925-07:00Tips for MapReduce with HadoopI found <a href="http://www.cloudera.com/blog/2009/05/18/10-mapreduce-tips/">these </a>nice set of tips for fine tuning MapReduce programs using Hadoop from the <a href="http://www.cloudera.com/">Cloudera </a>web site.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-36563631808508386572009-09-11T22:16:00.000-07:002009-09-11T22:50:23.186-07:00MSR Internship is over - Going back to IUToday I finished my 3 months internship at Microsoft research. It was quite a wonderful experiance for me, and I was able to accomplish most of my internship goals.<br /><br />At the beginning of my internship I was given the following goals for my internship.<br />Evaluate the usability of DryadLINQ for scientific analyses<br />– Develop a series of scientific applications using DryadLINQ<br />– Compare them with similar MapReduce implementations (E.g. Hadoop)<br />– Run above DryadLINQ applications on Cloud<br /><br />During the internship, I developed four DryadLINQ applications and optimized them for performance and also identified several improvements to the current DryadLINQ code base.<br /><br />I did a detailed performance analysis of the Cap3, HEP, Kmeans applications developed using DryadLINQ comparing them with Hadoop implementations of the same applications. Performance of the pair wise distance calculation application was compared with an MPI implementation of the same application. These findings were all included in the following two papers.<br /><a href="http://grids.ucs.indiana.edu/ptliupages/publications/DryadLINQ_for_Scientific_Analyses.pdf">DryadLINQ for Scientific Analyses</a><br /><a href="http://grids.ucs.indiana.edu/ptliupages/publications/MTAGS09-23.pdf">Cloud Technologies for Bioinformatics Applicaitons</a><br /><br />We (I and my colleague intern –Atilla Balkir) were able to deploy a Windows HPC cluster on GoGrid cloud. I was able to run Cap3 application on Cloud but other applications did not work due to the limitations of the GoGrid infrastructure.<br /><br />Overall we have the following conclusions regarding DryadLINQ runtime.<br /><ul><li>We developed six DryadLINQ applications with various computation, communication, and data access requirements<br />All DryadLINQ applications work, and in many cases perform better than Hadoop</li><li><span style="color:#cc0000;">We can definitely use DryadLINQ for scientific analyses</span> </li><li>We did not implement (find)<br />–Applications that can only be implemented using DryadLINQ but not with typical MapReduce</li><li>Current release of DryadLINQ has some performance limitations</li><li>DryadLINQ hides many aspects of parallel computing from user<br />Coding is much simpler in DryadLINQ than Hadoop (provided that the performance issues are fixed)</li><li>More simplicity comes with less control and sometimes it is hard to fine-tune</li><li>We showed that it is possible to run DryadLINQ on Cloud</li></ul><p>I got all the necessary support from my mentor (Nelson Araujo), Chirstophe, and the ARTS team @ MSR in accomplishing the objectives of my internship. I would also like to thank Dryad team at Silicon Valley for their dedicated support as well. Last but not least, the support from my advisor (<a href="http://www.informatics.indiana.edu/research/profiles/gcf.asp">Prof. Geoffrey Fox</a>) and the <a href="http://www.infomall.org/multicore/index.php/Main_Page">SALSA team</a> at pervasive technology labs was a tremendous encouragement to me.</p><p>Sunday we are planning to head back to Indiana with a two week old baby - Our small miracle - in our hands.</p><p></p>Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-45133089365905026712009-09-07T21:04:00.001-07:002009-09-07T21:45:00.473-07:00DryadLINQ for Scientific AnalysesI spent the last 3 months at Microsoft Research as an intern doing research on DryadLINQ. Our goal (myself and a another intern - <a href="http://www.cs.uchicago.edu/people/soner">Atilla Soner Balkir</a>) was to evalute the usability of DryadLINQ for scientific applications.<br /><br />We selected a series of scientific applications and developed DryadLINQ programs for those applications, and evaluated their performances. We compared the performance of the DryadLINQ applicaitons against Hadoop and in some cases MPI versions of the same applications.<br /><br />We identified several improvments to DryadLINQ and its software stack, and found workarounds to these inefficienies and was able to run most applicaitons with 100% CPU utilizations.<br /><br />We compiled a paper including our findings regarding DryadLINQ and submitted it for the eScience09 conference. You can find a draft of this technical paper <a href="http://grids.ucs.indiana.edu/ptliupages/publications/DryadLINQ_for_Scientific_Analyses.pdf">here</a>.<br /><br />Hope this will be usefull to some of you who are developing applications using DryadLINQ.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-71348368718436579822009-07-13T11:43:00.001-07:002009-07-13T11:44:32.638-07:00Microsoft released Dryad and DryadLINQ for Academic Use<a href="http://research.microsoft.com/en-us/collaboration/tools/dryad.aspx">http://research.microsoft.com/en-us/collaboration/tools/dryad.aspx</a>Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-44069823694164978632009-06-19T00:33:00.000-07:002009-06-19T00:38:43.446-07:00High Performance Parallel Computing with Clouds and Cloud TechnologiesWe compiled the latest results/findings of our research as a paper and submitted to CloudComp2009.<br />Following is the abstract of the paper.<br /><br />Infrastructure services (Infrastructure-as-a-service), provided by cloud vendors, allow any user to provision a large number of compute instances fairly easily. Whether leased from public clouds or allocated from private clouds, utilizing these virtual resources to perform data/compute intensive analyses requires employing different parallel runtimes to implement such applications. Among many parallelizable problems, most “pleasingly parallel” applications can be performed using MapReduce technologies such as Hadoop, CGL-MapReduce, and Dryad, in a fairly easy manner. However, many scientific applications, which require complex communication patterns, still require optimized runtimes such as MPI. We first discuss large scale data analysis using different MapReduce implementations and then, we present a performance analysis of high performance parallel applications on virtualized resources.<br /><br />You can find the draft of the paper <a href="http://grids.ucs.indiana.edu/ptliupages/publications/cloudcomp_submission.pdf">here</a>. <a href="http://grids.ucs.indiana.edu/ptliupages/publications/cloudcomp_submission.pdf"></a>Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-50829462075629913522009-05-14T11:42:00.000-07:002009-05-14T11:46:34.962-07:00How to control the number of tasks per node when you run your jobs in Hadoop cluster?One of my colleague asked me the following question.<br /><br /><span style="font-weight: bold;">How to control the number of tasks per node when you run your jobs in Hadoop cluster?</span><br /><br />We can do this by modifying the hadoop-site.xml. However, the exact xml for the properties are there in hadoop-default.xml.<br /><br />So here is the method.<br /><br />Modify the $HADOOP_HOME/conf/hadoop-site.xml and add the following lines.<br /><property><name></name></property><br /><span style="font-family:courier new;"><property></span><br /><span style="font-family:courier new;"> <name>mapred.tasktracker.map.tasks.maximum</name></span><br /><span style="font-family:courier new;"> <value>8</value></span><br /><span style="font-family:courier new;"></property></span><br /><br /><span style="font-family:courier new;"><property></span><br /><span style="font-family:courier new;"> <name>mapred.tasktracker.reduce.tasks.maximum</name></span><br /><span style="font-family:courier new;"> <value>2</value></span><br /><span style="font-family:courier new;"></property></span><br /><br />You can find these properties in Hadoop-default.xml, but better not modify them there.<br />Instead, copy the properties to hadoop-site.xml and change the value. Then the default values will be overridden by the properties in the hadoop-site.xml.<br /><br /><span style="font-family:courier new;">JobConf.setNumMapTasks()</span>. Is to define how many map tasks Hadoop should execute for the entire job. This simply determines the data splitting factor.<br /><br />With all three parameters we can control the Hadoop's job parallelism better.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-52361180969692771982009-05-13T16:35:00.001-07:002009-05-14T11:47:42.078-07:00High Energy Physics Data Analysis using Microsoft DryadA demo of the Dryad version of the HEP data analysis can be found <a href="http://cs.indiana.edu/%7Ejekanaya/papers/demos/hep_demo.html">here</a>.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-70055663834882104852009-04-05T09:16:00.000-07:002009-05-14T11:47:26.265-07:00Classes of MapReduce Applications and Different RuntimesIn my experiance with MapReduce runtiems and the applications, I have noticed few classes of applications where the users can benifit from the available MapReduce runtimes. Let me list them below with their common characteristics.<br /><br /><span style="font-weight: bold;">1. Pleasingly Parallel Applications.</span><br />E.g. Processing a set of bilogy data files using <a href="http://pbil.univ-lyon1.fr/cap3.php">Cap3 </a>program.<br /><br />This could be the most common class of parallel applications that many users needs. The common characteristic is the application of a function or a program on a collection of data files producing another set of data files. Processing a set of medical images is also another type of similar application.<br /><br />When using MapReduce runtimes for these type of applications, we can simply use "map only"<br />mode of the runtimes. For example, in Hadoop we can use 0 reduce tasks as follows.<br /><span style=";font-family:courier new;font-size:85%;" ><br />JobConf jc = new JobConf(getConf(), Cap3Analysis.class);<br />jc.setNumReduceTasks(numReduceTasks);</span><br /><br />I impletented the "map only" support for CGL-MapReduce as well and it greatly increase the usability of the runtime to another large class of parallel applications.<br /><br />More information about Cap3 and the performance of different runtimes such as Hadoop, Dryad, and CGL-MapReduce can be found in the <a href="http://www.cs.indiana.edu/%7Ejekanaya/cglmr.html">CGL-MapReduce web page</a>.<br /><br />This class of applications also suite best for the <a href="http://www.microsoft.com/azure/default.mspx">Azure </a>applications written using <a href="http://msdn.microsoft.com/en-us/library/dd179341.aspx">Worker Role</a> where a set of workers are assinged to process a computation tasks available in a Queue.<br /><br /><span style="font-weight: bold;">2. Typical MapReduce Applications</span><br /><br />E.g. High Energy Physics Data Analysis, where a set of analysis functions are executed on each data file of a collection of data files during the "map" stage producing a collection of intermediate histograms and in the "reduce" stage these histograms are merged together to produce the final histogram.<br /><br />Histogramming words and Distributed Grep are very common examples used in most tutorials.<br /><br />This class utilizes complete execution model of MapReduce. The application of the "reduce" stage and the function performed by that stage depend on the "<span style="font-style: italic;">associative</span>" and "<span style="font-weight: bold;">commutative</span>" properties of the application itself.<br /><br /><span style="font-weight: bold;">3. Iterative MapReduce Applications.</span><br /><br />This is a complex class of applications where we need to apply mutliple stages of MapReduce computations to accomplish the data analysis task. Kmeans clustering is a simple application of this nature, where each iteration of MapReduce cluster a set of data points to a given number of cluster centers. The next iteration takes the previous cluster centers as the input and calculate another set of cluster centers minimizing the difference.<br /><br /><span style="font-style: italic;">Matrix Multiplication</span> and<span style="font-style: italic;"> Multi Dimensional Scaling</span> aretwo more similar applications of this class and many machine learning algorithms also fit to this class of applications.<br /><br />However, the applicability of the current runtimes such as Hadoop and Dryad for this class of applications is questionable. According to our results shown in the following graph we can see the overhead added by runtimes such as Dryad and Hadoop for such data analysis. Also note how the light weight MapReduce runtime -CGL-MapReduce - and its support for iterative applications have enabled it to produce close results with MPI.<br /><br /><div style="text-align: center;"><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbjFfwbr8-mgd02xJQw9EMDEYVSDES9JwbZApjH75s_0JBEGZb6VQKSsSJcQ_StDq-0mQ2-VfgBddc4s_ZPb0qGDyoXltJqF2VtIO551QJdLKkxCrPtOfk7mF69BLdIz26p2CuhA/s1600-h/kmeans_perf_dry_had_cgl_mpi.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 215px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbjFfwbr8-mgd02xJQw9EMDEYVSDES9JwbZApjH75s_0JBEGZb6VQKSsSJcQ_StDq-0mQ2-VfgBddc4s_ZPb0qGDyoXltJqF2VtIO551QJdLKkxCrPtOfk7mF69BLdIz26p2CuhA/s320/kmeans_perf_dry_had_cgl_mpi.png" alt="" id="BLOGGER_PHOTO_ID_5321293765275881906" border="0" /></a>Figure 1. Performance of different parallel run times in executing Kmeans Clustering<br /></div><div style="text-align: center;"><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPkdI8RDI5897BZuzmBN-iuTPjqsOFGmacV6nxagTDjSDjZNN8KyUuNEtPX8G0EXkPUMtqoNgGRz_sbNd61z7W24EKIgN8f4pFjOweMf6vh48PkVWlSoAU0oYTIVotQpyDqPmqdg/s1600-h/kmeans_oh_dry_had_cgl_mpi.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 215px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPkdI8RDI5897BZuzmBN-iuTPjqsOFGmacV6nxagTDjSDjZNN8KyUuNEtPX8G0EXkPUMtqoNgGRz_sbNd61z7W24EKIgN8f4pFjOweMf6vh48PkVWlSoAU0oYTIVotQpyDqPmqdg/s320/kmeans_oh_dry_had_cgl_mpi.png" alt="" id="BLOGGER_PHOTO_ID_5321293876069903282" border="0" /></a>Figure 2. Overhead of different parallel run times in executing Kmeans Clustering<br /></div>Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com3tag:blogger.com,1999:blog-36948928.post-35700748885401947912009-04-05T09:10:00.000-07:002009-05-14T11:48:11.840-07:00Simple PasswordLess SSHTutorialI was configuring Xen virtual machines to run MPI programs. Running MPI,<a href="http://hadoop.apache.org/core/">Hadoop</a>, or even <a href="http://www.cs.indiana.edu/%7Ejekanaya/cglmr.html">CGL-MapReduce</a> require password less login between the nodes of a cluster.<br /><br />It is the same old steps that we need to perform, but found this <a href="http://blogs.translucentcode.org/mick/archives/000230.html">simple tutorial</a> which list the process very succinctly.<br /><br />To get MPI to work you also need to add the following line to your bash (or your shell) profile.<br /><br />export LAMRSH="ssh -x"Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-34902616350327018852009-02-27T22:53:00.000-08:002009-02-27T23:26:57.333-08:00Eucalyptus without a DNSIt has been a while that I last blog relating to my research. I went to my home country (Sri Lanka) for the winter break and came back in mid January. After coming here I just realized that I need a big vacation again :). I had to finish a lot of pending work and now back on track and I can blog again. Still it is 1.56 am in the morning :)<br /><br />I just got a set of Xen VMs running with <a href="http://eucalyptus.cs.ucsb.edu/">Eucalyptus </a>after a few hours of debugging to fix the following warning that I got when I try to ssh between the VM instances.<br />-----<br /><br />get_socket_address: getnameinfo 8 failed: Name or service not known<br />userauth_hostbased: cannot get local ipaddr/name<br />-----<br /><br />Actually, it has nothing to do with the VM usage but something I have missed in configuring host based authentication using ssh.<br /><br />Since it could be helpful to someone else, let me describe the problem and the solution.<br />I was trying to start a set of VM instances using a Eucalyptus cloud that has been setup in a iDataplex cluster here at Indiana. I was able to follow the guidelines and start the VMs.<br /><br />Then my goal is to run a set of MPI applications on these VMs. To get MPI working, I need to have password less login between the nodes. As I have previously configured this in the VM image things worked fine but it gave the above warning when I try login between nodes, which then stops MPI daemons from starting.<br /><br />When I used these VM images with Eucalyptus public cloud every VM instance got assigned a static IP. So, setting up any DNS was not required. This time, however, we use only one static IP for one of the nodes of the VM instances while the rest have dynamic IPs. Also there is no DNS setup for these VM instances. This causes the above warning when I try to login between the nodes using ssh.<br /><br />Joe Rinkovsky a Unix Systems Specialist @ Indiana explained me the fix. I need to add (IP , name) pairs to /etc/hosts file of the head node of the VM instances. The solution is explained <a href="http://users.telenet.be/mydotcom/howto/linux/sshpasswordless.htm">here</a> under "make sure name resolution works" which I haven't noticed till today :).Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-14381987452233699602008-11-17T11:00:00.000-08:002008-11-17T11:05:28.959-08:00MapReduce for Data Intensive Scientific AnalysesMost scientific data analyses comprise analyzing voluminous data collected from various instruments. Efficient parallel/concurrent algorithms and frameworks are the key to meeting the scalability and performance requirements entailed in such scientific data analyses. The recently introduced MapReduce technique has gained a lot of attention from the scientific community for its applicability in large parallel data analyses. Although there are many evaluations of the MapReduce technique using large textual data collections, there have been only a few evaluations for scientific data analyses. The goals of this paper are twofold. First, we present our experience in applying the MapReduce technique for two scientific data analyses: (i) High Energy Physics data analyses; (ii) Kmeans clustering. Second, we present CGL-MapReduce, a stream based MapReduce implementation and compare its performance with Hadoop.<br /><br />The <a href="http://www.cs.indiana.edu/%7Ejekanaya/papers/eScience-final.pdf">paper</a> accepted for the eScience 2008.Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0tag:blogger.com,1999:blog-36948928.post-83540932164794317042008-10-10T12:38:00.000-07:002008-10-10T13:05:24.738-07:00Multi Dimensional Scaling (MDS) using MapReduce<div style="text-align: justify;"> <span style="font-size:100%;"><span style="font-family:arial;">As part of my ongoing research on various parallelization techniques, I implemented a MapReduce version of a MDS program. My colleague </span><a style="font-family: arial;" href="http://grids.ucs.indiana.edu/ptliupages/publications/hpcsApril12-08.pdf">Seung-Hee Bea</a><span style="font-family:arial;"> has developed the in memory implementation of MDS using C# which runs on a multi-core computers. As the first step in converting that algorithm to a MapReduce implementation, I implemented a single threaded version of the same program in Java (since I am planning to use CGL-MapReduce as my MapReduce runtime). I ran this program to reduce 1024 four dimension (4D) data set to a 3D data set and measured the execution time of the different parts of the program. The result showed that</span><span style="font-weight: bold;font-family:arial;" > 97.7%</span><span style="font-family:arial;"> of the overall execution time of this program is used for matrix multiplication (both square matrices and matrix and vector multiplications).</span><br /></span></div><span style="font-size:100%;"><br /></span><div style="text-align: justify;"><span style="font-size:100%;"><span style="font-family:arial;"> This observation motivated me to first implement the matrix multiplication in MapReduce and parallelize the matrix multiplication part of the entire computation using MapReduce technique. I developed both square matrix and matrix and vector multiplication algorithms using CGL-MapReduce and incorporated these to the MDS algorithm.</span><br /><br /><span style="font-family:arial;"> I tested this MapReduce version of the MDS program using 1024 and 4096 points data sets (4 dimensional data reduced to 3D data) and obtained the following results. The data I used has a predefined structure and the results show the exact structure of the data. In addition the results also tally with the results of the single threaded program. So I can reasonably assume that the program performs as expected. The program performed the MDS on </span><span style="font-weight: bold;font-family:arial;" >1024</span><span style="font-family:arial;"> data set in nearly </span></span><span style="font-weight: bold;font-family:arial;font-size:100%;" >11 minuites</span><span style="font-size:100%;"><span style="font-family:arial;"> and the </span><span style="font-weight: bold;font-family:arial;" >4096</span><span style="font-family:arial;"> data set in </span></span><span style="font-weight: bold;font-family:arial;font-size:100%;" >2 hours and 40 minuites</span><span style="font-size:100%;"><span style="font-family:arial;">.</span></span><br /></div><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsu02GsOBeG40q97K2aXs-clOnB19X2p5FySKRA8z5vLCaNZTXUvtBW9gNK2x2JDgU_qnFqAwZOZsRElmnZal6688vjMjTwp5VqzDgk8rG9sxTfng1WcR-jUZw4AeCbvuHIAYBkg/s1600-h/GD_4D_496_1-24_all.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsu02GsOBeG40q97K2aXs-clOnB19X2p5FySKRA8z5vLCaNZTXUvtBW9gNK2x2JDgU_qnFqAwZOZsRElmnZal6688vjMjTwp5VqzDgk8rG9sxTfng1WcR-jUZw4AeCbvuHIAYBkg/s320/GD_4D_496_1-24_all.png" alt="" id="BLOGGER_PHOTO_ID_5255612879218945634" border="0" /></a><br /><div style="text-align: center;"><span style="font-weight: bold;font-size:85%;" ><span style="font-family:arial;">Figure 1. Resulting clusters of Multi Dimensional Scaling. Left hand side images shows the clusters visualized using </span><a style="font-family: arial;" href="http://www.cats.rwth-aachen.de/software/meshing/meshview">MeshView</a><span style="font-family:arial;"> software and the right hand side images illustrate the predefined structure expected in the data set using lines drawn on the same image.</span></span><br /></div><br /><span style="font-weight: bold;font-family:arial;" >Memory Size Limitation</span><br /><br /><div style="text-align: justify;"><span style="font-family:arial;"> The current MapReduce implementation of the MDS algorithm handles only the matrix multiplication in the MapReduce style. However, I believe that we can come up with a different algorithm which will allow most of the computations performed in MDS to be performed using MapReduce. This will reduce the amount of data that needs to be held in-memory by the master program allowing the MDS program to handle larger problems utilizing the total memory available in the distributed setting.</span><br /></div>Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com1tag:blogger.com,1999:blog-36948928.post-28546968542207682962008-08-20T17:53:00.000-07:002008-08-26T20:46:25.902-07:00Hadoop as a Batch Job using PBSDuring my previous data analyses using Hadoop and CGL-MapReduce I had to use the compute resources accessible via a job queue. For this purpose I used the Quarry cluster @ Indiana University which support batch job submissions via <a href="http://en.wikipedia.org/wiki/Portable_Batch_System">Portable Batch System</a>(PBS). I contacted one of the system administrators of the Quarry (George Wm Turner) and he point me to the Hadoop On Demand (<a href="http://hadoop.apache.org/core/docs/r0.16.4/hod.html">HOD</a>) project of Apache which mainly try to solve the problem that I am facing.<br /><br />We gave HOD a serious try but could not get it working the way we wanted. What we tried was to install HOD in set of nodes and let users to use it on demand via a job queue. This option simply did not work for multiple users, since the configurations options for the file system overlap between the users causing only one user to use Hadoop at a given time.<br /><br />With this situation, I decided to give it a try to start Hadoop dynamically using PBS. The task the script should perform is as follows.<br /><br />1. Identify the master node<br />2. Identify the slave nodes<br />3. Update $HADOOP_HOME/conf/masters and $HADOOP_HOME/conf/slaves files<br />4. UPdate the $HADOOP_HOME/conf/hadoop-site.xml<br />5. Cleanup any hadoop file system specific directories created in previous runs<br />6. Format a new Hadoop Distribtued File System (HDFS)<br />7. Start Hadoop daemons<br />8. Execute the map-reduce computation<br />9. Stop Daemons<br />10. Stop HDFS<br /><br />Although the list is bit long, most of the tasks are straigt forward to perfom in a shell script.<br /><br />The first problem I face is finding the IP addresses of the nodes. PBS passes this information via the variable PBS_NODEFILE. However, in a multi-core and multi-processor systems the PBS_NODEFILE contains multiple entries of the same node depending on the numebr of processors we requested in each node. So I have to find the "SET" of IP addresses eliminating the duplicates. Then have to update the configuration files depending on this information. So I decided to use a Java program to do the job (with my shell script knowledge I could not find an easy way to do a "SET" operation)<br /><br />Once I have this simple Java file to perform the steps 1 -4 the rest is straightforward. Here is my PBS script and I the link after the script will show you the simple Java program which update the configuration files.<br /><br />----------------------------------------------------------<br /><span style="font-family:courier new;">#!/bin/bash</span><br /><span style="font-family:courier new;">#PBS -l nodes=5:ppn=8</span><br /><span style="font-family:courier new;">#PBS -l walltime=01:00:00</span><br /><span style="font-family:courier new;">#PBS -N hdhep</span><br /><span style="font-family:courier new;">#PBS -q hod</span><br /><br /><span style="font-family:courier new;">java -cp ~/hadoop_root/bin HadoopConfig $PBS_NODEFILE ~/hadoop-0.17.0/conf</span><br /><br /><span style="font-family:courier new;">for line in `cat $PBS_NODEFILE`;do</span><br /><span style="font-family:courier new;"> echo $line</span><br /><span style="font-family:courier new;"> ssh $line rm -rf /tmp/hadoop*</span><br /><span style="font-family:courier new;">done</span><br /><br /><span style="font-family:courier new;">var=`head -1 $PBS_NODEFILE`</span><br /><span style="font-family:courier new;">echo $var</span><br /><br /><span style="font-family:courier new;">ssh -x $var ~/hadoop-0.17.0/bin/hadoop namenode -format</span><br /><span style="font-family:courier new;">ssh -x $var ~/hadoop-0.17.0/bin/start-dfs.sh</span><br /><span style="font-family:courier new;">sleep 60</span><br /><span style="font-family:courier new;">ssh -x $var ~/hadoop-0.17.0/bin/start-mapred.sh</span><br /><span style="font-family:courier new;">sleep 60</span><br /><span style="font-family:courier new;">ssh -x $var ~/hadoop-0.17.0/bin/hadoop jar ~/hadoop-0.17.0/hep.jar hep ~/hadoop_root /N/dc/scratch/jaliya/alldata 40 2</span><br /><span style="font-family:courier new;">ssh -x $var ~/hadoop-0.17.0/bin/stop-mapred.sh</span><br /><span style="font-family:courier new;">ssh -x $var ~/hadoop-0.17.0/bin/stop-dfs.sh</span><br />-------------------------------------------------------------------<br />[<a href="http://www.cs.indiana.edu/%7Ejekanaya/HadoopConfig.java">HadoopConfig.java</a>]<hadoopconfig.java><br /><br />As you can see in the script, I use ssh to log into each node of the cluster and perform the cleaning up of the HDFS directories. Then I use SSH to login to the master node to start the Hadoop daemons.<br /><br />Next comes the actual execution of the data analysis task, which I have coded in the hep.jar.<br />After the MapReduce computation is over the rest of the commands simply stop the daemons.<br /><br />This is method has more flexibility to the user and requires no changes for the batch job scheduling system. It also serves as an easy opti<div class="cssButtonOuter"><div class="cssButtonMiddle"><div class="cssButtonInner"><a>Save Now</a></div></div></div>on when the number of MapReduce computations are smaller than the other batch jobs.<br /><br />However, if the cluster is dedicated to run MapReduce computations, then everybody starting and stopping a HDFS file system does not make sense. Ideally the Hadoop should be started by the adminitrators and then the users should be allowed to simply execute the MapReduce computations on it.<br /><br />Your comments are welcome!<br /></hadoopconfig.java>Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com3tag:blogger.com,1999:blog-36948928.post-12411374643840212572008-08-20T17:51:00.001-07:002008-08-20T17:53:07.746-07:00High Energy Physics Data Analysis Using Hadoop and CGL-MapReduceAfter the previous set of tests wtih parallel Kmeans clusting using CGL-MapReduce, Hadoop and MPI I shift the direction of testing to another set of tests. This time, the test is to process large number (and volume of) High Energy Physics data files and produce a histogram of interesting events. The amount of data that needs to be processed is 1 terabyte.<br /><div style="text-align: justify;"><br />Converting this data analysis into a MapReduce version is straigt forward. First the data is split into managable chunks and each map task process some of these chunks and produce histograms of interested events. Reduce tasks merge the resulting histograms producing more concentrated histograms. Finally a merge operation combine all the histograms produced by the reduce tasks.<br /><br />I performed the above test by incresing the data size on a fixed set of computing nodes using both CGL-MapReduce and Hadoop. To see the scalability of the MapReduce approach and the scalability of the two MapReduce implementations, I performed another test by fixing the amount of data to 100GB and varying the number of compute nodes used. Figure 1 and Figure 2 shows my findings.<br /><div style="text-align: center;"><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3I9wmh4Z4CaQlAjItmoT25hqjTBkQY7wBGeh6YTZjwKT7AoHO9dc8v2SlNoa9ew89ZhWz9EKey0C8umiXqyxJu7t9tAkV3b0MyWVeRgHlEtJYh7RVj0zJOiHhN70PvaKUQ572NA/s1600-h/es_hep.PNG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3I9wmh4Z4CaQlAjItmoT25hqjTBkQY7wBGeh6YTZjwKT7AoHO9dc8v2SlNoa9ew89ZhWz9EKey0C8umiXqyxJu7t9tAkV3b0MyWVeRgHlEtJYh7RVj0zJOiHhN70PvaKUQ572NA/s320/es_hep.PNG" alt="" id="BLOGGER_PHOTO_ID_5236767415713761698" border="0" /></a>Figure 1. HEP data analysis, execution time vs. the volume of data (fixed compute resources)<br /></div><div style="text-align: center;"><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFaQm7ZoqbI61uXZrjCG1KgVFPQ7_iGewJ20EQVzi2YCGyvOJMsSzpgPFayB78ZSNUOP4CRaIq98xphbup2zdBX_zlIujY0tggHlp9ahFI-1WiFUzVcd6gcuXT1oHSaZzOt_qBdg/s1600-h/es_scale.PNG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFaQm7ZoqbI61uXZrjCG1KgVFPQ7_iGewJ20EQVzi2YCGyvOJMsSzpgPFayB78ZSNUOP4CRaIq98xphbup2zdBX_zlIujY0tggHlp9ahFI-1WiFUzVcd6gcuXT1oHSaZzOt_qBdg/s320/es_scale.PNG" alt="" id="BLOGGER_PHOTO_ID_5236767560231937938" border="0" /></a>Figure 2. Total time vs. the number of compute nodes (fixed data)<br /></div><br />Hadoop and CGL-MapReduce both show similar performance. The amount of data accessed in each analysis is extremely large and hence the performance is limited by the I/O bandwidth of a given node rather than the total processor cores. The overhead induced by the MapReduce implementations has negligible effect on the overall computation.<br /><br />The results in Figure 2 shows the scalability of the MapReduce technique and the two implementations. It also shows how the performance increase obtained by the parallelism diminshes after a certain number of computation node for this particular data set.<br /></div>Jaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.com0