Implementation and Results of Hypothesis Testing from the C

Implementation and Results of Hypothesis Testingfrom the C I Parallel Benchmark SuiteBrian V anV oorst,Rakesh Jha,Luiz Pires,Mustafa MuhammadHoneywell Technology Center3660Technology DriveMinneapolis,MNvanvoors,jha,pires,muhammad@AbstractThis paper describes the implementation of the hypothe-sis testing benchmark,one of ten kernels from the C I(Com-mand,Control,Communications and Intelligence)ParallelBenchmark Suite(C IPBS).The benchmark was imple-mented and executed on a variety of parallel environments.This paper details the run times obtained with these imple-mentations,and offers an analysis of the results.1.IntroductionThe C I(Command,Control,Communications and In-telligence)Parallel Benchmark suite is sponsored by RomeLaboratory and developed by the Honeywell TechnologyCenter in collaboration with ALPHA TECH and the Univer-sity of Minnesota.The benchmark methodology is based onpen and paper specifications and closely follows the effortsof the NAS parallel benchmarks[2]and those developed bythe PARKBENCH committee[4].The main reason for de-veloping a new set of benchmarks is that C I applicationsare significantly different from the scientific computationsaddressed by the existing benchmarks.The suite consistsof ten benchmarks,and for each one we have developed afunctional specification and serial code.To start,five ofthe ten benchmarks are being implemented on several par-allel platforms[5].An overview of all of the benchmarksappears in[6].This paper details the implementation andresults of one benchmark,hypothesis testing.Hypothesistesting is a widely applicable problem that has not beenaddressed yet by the high performance computing commu-nity.This paper describes our approach to parallelization,presents the results to-date,and offers a starting point forother who wish to implement this benchmark.the major portions of hypothesis testing here.Our descrip-tion follows the pseudocode outlined in Figure1.In the benchmark there are a series of radar“frames”which contain the x,y coordinate pairs of measurements (blips)detected by radar.The series of frames are consecu-tive radar images of the same geographical area over a pe-riod of time.The purpose of the benchmark is to establish which measurements are for the same object in the differ-ent radar frames,thereby tracking the object.Three tracks, composed of radar returns,are shown in Figure2.Figure 2.Visual representation of tracksacross frames.The frames are processed in order of increasing time. The benchmark takes the existing hypothesized tracks from frame N and projects them forward for evaluation with frame N+1(Figure1,line2).A given projected track may be near several x,y measurements in frame N+1.For each measurement near the projected track,we use a gating func-tion(Figure1,line6).A gating function considers location, velocity,history,and probable error to compute an x,y area in which a track is likely to go in the future.An x,y point is said to“pass gating”for a track if it is within the region computed by the gating function.For each x,y measurement that passes gating a new track extension is hypothesized to exist and a new hypothesis is created(Figure1,line8).This increases the number of hypotheses that must be evaluated in future frames as shown in Figure3.Figure3.Exponential growth of hypotheses.Because of the exponential growth of hypotheses from frame to frame we must have a mechanism for discarding some hypotheses to conserve memory.Each track is given a score based on the properties of the hypothetical track that indicates the likelihood of it being true.The bench-mark specification requires that the benchmark keep the best250,000hypotheses in memory at all times.Therefore, we can discard(prune)any hypothesis that is not in this set (Figure1,line17).The best way to compute this threshold is to use one of the well known selection algorithms.At the end of the benchmark,the K best scoring hypotheses that do not share measurements are reported as being the true tracks,and the benchmark terminates(Figure1,line19).The benchmark allows for the possibility of objects go-ing undetected in a frame,or for false objects(clutter)to ap-pear.Clutter does not form tracks and only interferes with real radar data.Also,real radar data is“noisy”meaning that x,y coordinates may be slightly off from their“true”values. The result is that objects that travel in a straight line may not have exactly straight line x,y coordinates.4.Distributing Hypotheses for ParallelizationOne approach to parallelizing hypothesis testing is to exploit the fact that creation,evaluation and extension of individual hypotheses are completely independent of each other.This means,theoretically,that multiple processors can start with a set of hypotheses and can do all the com-putations on their local hypotheses(and the hypotheses’subsequent offspring hypotheses)without the need for any communication or synchronization during the processing of a frame.The upper bound of this parallelism is the number of initial hypotheses at the start of the benchmark.In the case of the inputfile for this benchmark,that number is277 and for general scenarios would range from200to600.We have implemented the benchmark on a wide range of message passing architectures and parallel environments. On the Paragon,we ran an MPI and NX version.On the Cray T3D,we ran an MPI and a Cray Get/Put version.We ran the MPI version on both SGI Challenge and a Power Challenge.We also ran the MPI version on a RS6000clus-ter and an IBM SP2,both featuring the590“wide”nodes. This combination of machines and tools gave us the chance to compare and contrast different message passing environ-ments,different interconnects with the same processor,and different machines in the same product line.It is important to remember that these are our sample im-plementations based on the specification.Other individuals are welcome to code their own implementation as long as it satisfies the specification.Our parallel implementations follows the distributed hy-potheses formulation described above.The root processor reads in the input,including all measurement frame data, and broadcasts it to the other processors.Timing begins af-ter the root processor has completed reading in the input, but before the input is broadcast to the other processors. To distribute the hypotheses,each processor simultaneously goes through the initial measurements in thefirst frame and picks out their measurements in a round-robin fashion.This starts the benchmark in a nearly balanced state(+/-one hy-pothesis)for each processor.The benchmark starts the“For each existing hypothesis track do”loop(Figure1,line2)on each processor,with each processor having approximately 1/p the amount of work to do(where p is the number of processors).Work continues in this parallel fashion until the prune hypotheses stage(Figure1,line17).Here a call is made to the randomized parallel selection algorithm[1],tofind the 250,000th best scoring hypotheses.If there are fewer than 250,000hypotheses across the processors the code returns immediately;otherwise,it returns the250,000th largest likelihood score.While selection is not expensive relative to the overall cost of computation,it is important to note that this stage becomes a synchronization barrier-no pro-cessor can proceed until the slowest processor has reached this stage.Once the likelihood score of the250,000th el-ement(the prune value)is selected and broadcasted,each processor discards any hypotheses it has with a score less than the prune value.After pruning,the code may invoke a load-balancing routine.Results have shown this application can be very sensitive to load imbalance,this issue is discussed in Sec-tion4.3.Following load balancing,the main loop incre-ments the frame count by one and the benchmark starts on the next frame’s data.As the computation continues, a processor will grow its initial hypotheses into the tens-of-thousands,pruning and redistributing them for load balanc-ing as necessary.Load balancing is an important step in the benchmark. Pruning forms a synchronization point,and all processors block until the slowest processor reaches the pruning stage. If that processor is twice as slow as the others,then effi-ciency will be cut in half.Classical approaches to load balancing,as they apply to hypothesis testing,can be summarized as follows:1.Each processor decides if it is sending or receiving work(based on a chosen metric such as number of hypotheses orrun time).2.A mapping is made between senders and receivers.3.Sending nodes send their extra hypotheses.4.Receiving nodes asynchronously receive the extra hypothe-ses.5.All nodes start processing their existing local hypotheses.6.Receiving nodes,whenfinished with local hypotheses,pro-cess the extra hypotheses received.Because the receive is done last,communication is over-lapped with computation.Receiving nodes should be the lightly loaded nodes thatfinish early,and therefore have time to process the extra work.While the principle is sound,the problem lies in cor-rectly determining which processors in the future will be lightly loaded and which will have excess work.A pro-cessor’s load cannot be determined only by the number of hypotheses it currently has,but must also consider future offspring.The cost of sending a hypothesis(about230 bytes)can be as much as the cost to compute it for one frame(about450FLOPs);however it is impossible to pre-dict what offspring hypotheses will be generated by a hy-pothesis.This makes it very tricky selecting hypotheses to move.To help illustrate the magnitude of the problem consider the following.The average number of measurements a hy-pothesis gates with is seven.Suppose in frame N,one (lucky)hypothesis gates withfifty measurements,making fifty new hypotheses.Now suppose in frame N+1thefifty-one hypotheses,all tracks going in the same direction,each gate with ten measurements,producingfive hundred and ten hypotheses.This is an order of magnitude more offspring than the expected amount.Figure3helps illustrate this.No-tice how much more offspring hypotheses B has than A.An additional50hypotheses(22,500FLOPS,11,500 bytes)are negligible in frame N+1;however,several frames later it could grow to be a big concern.This is compli-cated by the fact that allfive hundred and ten hypotheses may have nearly the same likelihood scores and could all be pruned in the next frame,resulting in a big loss of work. As this example illustrates,moving one hypothesis can ei-ther have a huge impact or be inconsequential,however this is not known until much later.Since the benchmark runs for only ten frames there is not much time to stabilize the load.Furthermore,static load balancing for the dataset is not allowed by the benchmark specification.Two measures of load have been tried with marginal suc-cess.Thefirst attempt to determine a processor’s load was based on the number of hypotheses it has.The goal was to make sure each processor had a nearly equal number of hy-potheses for each frame.At the end of a frame,processors with too many hypotheses sent hypotheses to the processors with too few.This was not successful if processors heldonto a hypothesis with“high yield”(meaning they gener-ated a lot of other hypotheses).When this happened,these processors remained overworked from frame to frame.One explanation is that there are not enough high yield hypothe-ses so that each node can have one,making load balancing difficult when scaling to larger numbers of nodes.A second measure of load was to record the time taken by each processor in the main loop,and after each frame de-termine which processor had worked more and which had worked less.Based on this information,overworked proces-sors would send some hypotheses to under-worked proces-sors in an attempt to improve the balance for future frames. This was not successful since it appears that being over-worked can be a transient state(due to pruning),with little consistency from one frame to the next.As seen in the results(Section5),load imbalance poses a serious problem to the scalability of this application.Other approaches to load balancing in hypothesis testing such as partial randomized redistribution,work sharing,and spec-ulative execution of the next frame may be considered in future implementations.Shared memory implementations may be able to avoid many of these problems.These ideas are discussed in[7].At the end of the benchmark each processor has several thousand hypotheses,and we wish tofind the K best mutu-ally exclusive hypotheses as previously described in Figure 1.Picking the best mutually exclusive hypotheses is per-formed with the following algorithm:Each processor sorts their hypotheses into descending orderbased on likelihood score.Processors form a comparison tree,with processor0at the root.For loop=1to kEach processor puts their best scoring hypothesis that is mutually exclusive with the previous k-1best hypotheses into the comparison tree.In log2(p) stages the k best scoring hypotheses is known to processor0.The root broadcasts the k best hypotheses data and the other processorswait for the broadcast.There are some subtleties to be noted.First,when a pro-cessor puts hypothesis j into the comparison tree it will only advance to the j+1hypothesis if either j is chosen as being best,or j is not mutually exclusive with the ones already chosen best.Otherwise,the processor will re-submit the same hypothesis j in the next round.Second,while a pro-cessor is waiting to receive the broadcast from node0,it can go through its list of hypotheses(j+1and greater)and eliminate those which are not mutually exclusive with those hypotheses already selected,thus overlapping communica-tion and computation.Details of implementations on individual target plat-forms and analysis follow next.Table1summarizes run times on the platforms,and Figure4shows a plot of the results.5.Results and Analysis1010010001248163264128 WallclocktimeinsecondsProcessing elementsHT Benchmark TimingsNX:OSF ParagonMPI:RS6000 ATM ClustMPI:SGI R4400 ChallengeMPI:SGI R8800 Power ChlMPI:T3DGet / Put:T3DMPI:IBM SP2MPI:OSF ParagonFigure4.Plot of run-times.All runs were made under dedicated conditions,either explicitly through dedicated time or via a batch queuing system that runs one job at a time.All MPI codes were compiled with the MPICH1.0.12library,and all sources were compiled with the local C compiler.Data is only re-ported for runs thatfit entirely in main memory.When there was sufficient memory the serial code was run for a single processor to establish a baseline for comparison against the parallel code.In all respects the machines were standard for their product line.Additional information on the machines used can be found in[7].The following observations can be made:The implementation does not scale well beyond32 processors.Closer examination shows that this is due to load imbalance.The problem of load balancing was discussed in Section4.3.The implementation is somewhat sensitive to changes in topology and communication performance.The NAS IBM SP2and the University of Minnesota IBM 590cluster have the same processors,operating sys-tem and memory configuration.Both machines were benchmarked in a dedicated mode.The SP2uses its custom high performance switch,while the cluster uses an A TM switch.The A TM switch has a theoret-ical peak performance of19.3MB/s,about two thirds the throughput of the SP2.We see that the cluster’s runs are5-10%slower than the SP2,suggesting this application is somewhat sensitive to communication costs.PE’s Cray T3D SGI Pow Chal Clust Paragon IBM590Clust (MPI)(MPI)(MPI)(MPI) 1329.02375.19184.31485.24100.8763.5953.4497.3551.221634.0334.4853.0021.0016.8128.4616.636413.3518.7110.3417.1712.09。