tag:blogger.com,1999:blog-36948928.post4712080789684313318..comments2023-11-05T01:36:13.893-08:00Comments on Open Notebook: Twister: Iterative MapReduceJaliya Ekanayakehttp://www.blogger.com/profile/12210985278265903305noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-36948928.post-83322902014718231352012-01-29T04:47:29.252-08:002012-01-29T04:47:29.252-08:00Hi,
I am currently comparing two implementations ...Hi,<br /><br />I am currently comparing two implementations of PageRank. The one from Twister and another one.<br />My concern with Twister - at the end of the calculation, if I sum up all Ranks, shouldn't I get SUM=1.<br /><br />Instead I get '0.15663257775113015'malchohttps://www.blogger.com/profile/05048352442014791670noreply@blogger.comtag:blogger.com,1999:blog-36948928.post-24154673936674750032011-04-10T21:28:40.783-07:002011-04-10T21:28:40.783-07:00To Elif,
First of all let me verify if the setup ...To Elif,<br /><br />First of all let me verify if the setup is correct. <br />1. Did you enable fault tolerance using JobConf.<br />jobConf.setFaultTolerance();<br /><br />2. Did you try an iterative applicaiton or a normal MapReduce applicaiton?<br /><br />Twister provide fault tolerance only for iterative applications. What it do is restart the failed iteration.Jaliya Ekanayakehttps://www.blogger.com/profile/12210985278265903305noreply@blogger.comtag:blogger.com,1999:blog-36948928.post-30354079211533218252011-04-10T21:19:05.317-07:002011-04-10T21:19:05.317-07:00Hi,
I've set twister 0.9 to our cluster in my...Hi,<br /><br />I've set twister 0.9 to our cluster in my school. I am trying to figure out the case where the fault tolerance kicks in. As far as I see, there is no fault tolerance for the case where one of the mappers die after starting the execution.The job just hangs. Is this correct? If so what is the fault tolerance that you talk about in the paper refers to?- By the way I tried 2,4 and 16 replicas of each split for a 16 node job. Still the job is not completed<br /><br />thanksUnknownhttps://www.blogger.com/profile/05647903196989547814noreply@blogger.comtag:blogger.com,1999:blog-36948928.post-12807723828803071872010-02-10T23:25:53.570-08:002010-02-10T23:25:53.570-08:00Just saw this mailing list discussion at Apache Ma...Just saw this mailing list discussion at Apache Mahout project regarding Twister.<br />http://old.nabble.com/Fwd:-Twister:-Iterative-MapReduce-td27535589.html<br /><br />They thought we are competing. Cool. <br />Just thought that I better clarify some of the issues they have mentioned related to Twister.<br /><br />1. No fault tolerance. <br />Yes, this is just in this release. It will be ready soon for iterative MapReduce applications.<br /><br />2. Clusters are small.<br />We have tried twister with clusters up to 32 nodes with 256 CPU cores. Soon we will try it on 768 core cluster. These can be small compared to Google, or Yahoo, but when we look at the spectrum of parallel applications, i would say these are modest clusters.Jaliya Ekanayakehttps://www.blogger.com/profile/12210985278265903305noreply@blogger.com