Instruction Replication for Reducing Delays due to Inter-PE Communication Latency

Instruction Replication for Reducing Delays due to Inter-PE Communication LatencyAneesh Aggarwal,Manoj FranklinAbstract—As feature sizes are becoming smaller,wire delays are becoming very critical.Clustering is a popular decentraliza-tion approach to reduce the impact of shrinking technologies on clock speed.In this approach,the centralized instruction window is replaced with multiple smaller windows,called clusters(PEs). The performance of these clustered processors depends on the amount of inter-PE communication and load imbalance incurred by the distribution algorithm used to distribute instructions among the PEs.In this paper,we investigate a novel approach of reducing the impact of inter-PE communication latency,while preserving good load balance.The basic idea is to selectively replicate instructions in those PEs where their results are re-quired.The replication is done based on heuristics that weigh the potential benefits of replication.We found that with instruction replication,the IPC of a clustered processor is significantly higher than that obtained without instruction replication and is within just8%of that of a superscalar configuration with a centralized instruction scheduler.Keywords:Clustered Processors,Instruction Replication, Interconnection Latency,Load balancing,Task assignmentI.I NTRODUCTIONThe shrinkage in MOS transistor feature sizes is affecting processor design in many ways.A major implication is that wire delays have become a critical design issue[1],[17],[24]. This effect is predominant in global wires because of their long lengths.A popular way to deal with the wire delay problem is to decentralize the critical resources in the processor chip. That is,build the processor as a collection of independent Processing Elements(PEs)[4],[5],[9],[10],[11],[13],[16], [17],[20],[22],such that very little communication occurs through slow global wires and most of the communication occurs through fast local wires.Apart from reducing the impact of wire delays on clock speed,decentralized designs are intrinsically energy-efficient because of smaller modules, fewer ports,and localized interconnects[23].Figure1shows a decentralized(or clustered)processor, where each PE has a dynamic scheduler(DS),a registerfile (RF),and a few functional units(FUs).Instructions from multiple PEs are issued independently,subject only to the availability of operand values.An interconnection network (ICN)connects the PEs together for supporting inter-PE communication.A common fetch and decode unit distributes dynamic instructions among the PEs based on an instruction distribution algorithm.Executed instructions are committed from the PEs in program order.Aneesh Aggarwal is with the ECE Department at Binghamton University, NY.Manoj Franklin is with the ECE Department at University of Maryland, College Park,MD.Fig.1.A Generic Clustered Processor with4PEsThe distribution algorithm plays a major role in deciding the performance,because it determines the amounts of inter-PE communication and load imbalance incurred.Many instruction distribution algorithms have been proposed and evaluated for clustered processors[2],[4],[5],[9],[10],[19],[14],[21]. There are two major problems with these algorithms:1)It is difficult to achieve a perfect instruction distributionbecause:reduction in inter-PE communication requires data-dependent instructions to be assigned to the same PE,whereas good load balancing requires instructions to be equally distributed among the PEs.An algorithm striving to reduce inter-PE communication generally suffers from load imbalance,and one striving for good load balance suffers from high inter-PE communication[2].Some algorithms compromise by assigning data-dependent instructions to the same PE and the data-independent ones to different PEs,and also by adapting the algorithm at run-time[9],[19].When compared toa centralized dynamic scheduler,these algorithms stillhave a significant room for improvement[2],[5].2)The algorithms suffer from lack of lookahead.This isbecause the dispatcher distributes the fetched instruc-tions solely based on the current state of the processor, frequently causing the producers of the source operands of an instruction to be assigned to distinct PEs. Because of these reasons,the IPC(instructions per cycle) obtained from a clustered processor tends to be significantly lower than that of a superscalar processor[2],[5],[9].In this paper,we investigate an approach of using correc-tion techniques to overcome the limitations of an instruction distribution algorithm.In particular,we focus on instruction replication,where a few key instructions are selectively repli-cated in the PEs where their results are required.Such aselective replication of instructions reduces the impact of inter-PE communication latencies.The replicated instructions are given a secondary status,in the sense that they only use the idle resources,thus maintaining the load balance wrought by the base distribution algorithm.We found that with instruction replication,for most of the benchmarks,the IPC is within 5%of that attainable with a hypothetical zero-latency inter-PE interconnect and within about8%of that attainable with a superscalar configuration.Our approach also outperforms previously proposed instruction distribution algorithms.The rest of the paper is organized as follows.In Section2 we discuss the related work and the motivation behind our ap-proach.Section3presents the basic idea,different techniques, and the implementation details of instruction replication.Sec-tion4gives the experimental setup,the results,and analysis of the results.Section5discusses enhancements to the basic techniques so as to further improve the IPC.Section6briefly discusses the impact of different hardware parameters on the performance of instruction replication.Section7discusses techniques to remove instruction replication from the critical path.Finally,Section8concludes with the majorfindings.II.B ACKGROUND AND M OTIVATION Clustered processors suffer from the inter-play of inter-PE communication and load balancing.Our approach starts with a base instruction distribution algorithm and then uses a correction technique to overcome the limitations of the algorithm.For instance,if the base algorithm gives good load balancing but not so low inter-PE communication,then the correction technique reduces the impact of inter-PE communi-cation.On the other hand,if the base algorithm gives reduced inter-PE communication but high load imbalance,then the correction technique reduces the impact of load imbalance. The correction technique will be different in the two cases.A.MotivationTo motivate this approach,we study3instruction distribu-tion algorithms—[5],which gives good load balance but high inter-PE communication1,first-fit[5],which gives reduced inter-PE communication but high load imbalance2, and dependence-based[10],which gives importance to both load imbalance and inter-PE communication3.For all three algorithms,we measured the IPCs obtained with an1-cycle crossbar interconnect,an zero-cycle interconnect(correspond-ing to the scenario when the correction techniques have eliminated the delays due to inter-PE communication),and a hypothetical clustered processor in which a ready instruction can use an issue slot from any of the PEs(corresponding to the removal of all the delays due to load imbalance).Finally, 1This algorithm distributes the fetched instructions equally among the various PEs,such that3consecutive instructions are assigned to each PE at a time.2This algorithm distributes the instructions to the same PE till itfills up and then distributes the instructions to the next PE and so on.3This algorithm assigns data-dependent instructions to the same PE and the data-independent ones to different PEs.If considerable load imbalance is encountered,it assigns the instructions to the least-loaded PEs without considering inter-PE communication.we also measured the IPCs with a hypothetical superscalar processor having the same overall instruction window size and issue width.Table I gives the IPCs for a collection of SPEC2000integer andfloating point benchmarks and a high IPC integer matrix multiplication(mm)bench-mark,using the experimental parameters of Table II.As seen in Table I,the IPCs from with zero-cycle interconnect latency,and that fromfirst-fit with perfect load balance,are very close to that with the superscalar configura-tion.On the other hand,the increase in IPC with perfect load balance for and with zero-cycle interconnect forfirst-fit,is marginal.This is expected because suffers from high inter-PE communication and very small load imbalance, andfirst-fit suffers from high load imbalance and very small inter-PE communication[2],[5].For the dependence-based algorithm,which gives importance to both load imbalance and inter-PE communication,the load imbalance incurred is higher than that with but smaller than that withfirst-fit,and the inter-PE communication incurred is higher than that withfirst-fit but smaller than that with[2],[5].Hence,although the base dependence-based algorithm performs better thanandfirst-fit,when either interconnect latency or load imbalance is reduced to zero,the IPC obtained is not as high as that obtained with zero-cycle interconnect as well as zero load imbalancefirst-fit.Another important observation is that when using a correction technique,the potential IPC that can be obtained from clustered processors is significantly higher than that obtained without the correction techniques.In this paper,we focus on a correction technique to reduce the impact of inter-PE communication latency.As seen in Table I,such a technique is most advantageous when used with a base algorithm giving the best load balancing.Therefore,our objective is to develop a run-time correction technique that reduces the impact of inter-PE communication latency for a load balancing-oriented base algorithm.B.Related WorkAn approach slightly resembling the approach of instruc-tion replication(at a macro level)is used in the datascalar architecture in[8].In this architecture,multiple autonomous PEs,tightly coupled with a fraction of the system’s physical memory,execute the same program.When a data value is loaded,it is broadcasted to all of the PEs,using a fast interconnect network.Another approach used to reduce inter-PE communication in clustered processors is proposed in[18]. In this approach,value prediction is used to avoid the delay associated with inter-PE communication.The predicted value is then validated locally where it is produced and communi-cation is experienced only in the case of misprediction.Kuras et.al.[15]propose value cloning to reduce inter registerfile communication for VLIW processors.Alex et.al.[3]propose compile-time instruction replication technique for clustered VLIW processors.III.I NSTRUCTION R EPLICATION TO R EDUCE THE I MPACT OF I NTER-PE C OMMUNICATIONIn this section,we discuss the basic idea of instruction replication,different scenarios where replication can help,andFirst-fit Dep-based Super 0-cycle0-cycle0-cyclelatency latency latency2.70 2.13 2.53vpr 1.83 1.85 1.85 2.20 2.01 2.11 2.231.83 1.71 1.77crafty 1.94 1.99 2.15 2.73 2.23 2.58 2.791.72 1.56 1.63vortex 3.22 3.22 3.25 3.51 3.29 3.48 3.593.63 3.13 3.38twolf 2.46 2.50 2.53 2.77 2.58 2.70 2.842.55 2.35 2.41mesa 2.16 2.17 2.20 2.47 2.26 2.34 2.523.64 3.26 3.48Either all3instructions(,,)have been assigned to different PEs(Figure2(i)),or only and haveReplicating yScenario 1(i) Replicating y(ii)PE 2PE 0dFig.2.Possible Scenarios for the assignment of data-dependent instructions when dispatchingbeen assigned to the same PE(Figure2(ii)).In either case, replicating in’s PE will help avoid the delay in’s issue due to inter-PE communication.In Figure2(i),when forwards the result,both and get their operands simultaneously. In Figure2(ii),gets its operands even before,potentially reducing the delay in’s issue.Scenario2:4Figure2considered only a single operand for each instruction. With multiple operands instructions,the scenarios indepen-dently apply to each of the operands of the instructions.C.Replication TechniquesNext,we propose two different replication techniques forreducing the impact of inter-PE communication latency.In these techniques,replication is done only for the instructions that are in the dispatch(steering)stage.Once an instructionis physically dispatched(steered)to a PE,it is not replicated.1)Myopic Replication(MyRep):In this technique,repli-cation of instructions is performed by considering only the benefit to instructions that are currently being dispatched.That is,if both the consumer and producer instructions are present in the current dispatch set and are being assigned to different PEs,then the producer is replicated in the consumer’sPE.Figure3shows a simplified example for MyRep.This example uses4PEs and the load balance–oriented instruction distribution algorithm.To understand the benefitsof instruction replication,instruction assignment and data forwarding are shown in Figure3both with and without instruction replication.The darker arrows indicate inter-PEcommunication and the lighter ones intra-PE communication. In thisfigure,instructions to have been already dispatched to the PEs.In the next cycle instructions to are evenly distributed among all the PEs using the base distri-bution algorithm.But,at dispatch time,the data-dependencies (shown in the middle of thefigure)are recognized,and some replications are performed.For the dependency, is not replicated because we do not replicate an already dispatched instruction.For the dependency,is replicated in the PE assigned to(PE1),avoiding the inter-PE communication delay from(PE0)to,provided the value from is simultaneously forwarded to both and. Similarly,for the dependency,is replicated in PE3,avoiding the delay from to.For the and dependencies,both instructions are already assigned to the same PE and hence there is no need for replication. For the dependency,is not replicated because is dependent on which is in the same PE,and in this technique we do not replicate chains of instructions.2)Look-ahead Replication(LaRep):The MyRep technique does not take into account what will be fetched in the future; therefore suffering from lack of lookahead,causing inter-PE communication related delays for future instructions.Hence, we investigate a replication technique that benefits not only the instructions in the current set of dispatched instructions, but also those in the future sets.To this end,run-time history of instructions is dynamically collected(by the hardware).When an instruction commits, the PEs where the instruction’s result is required is recorded using the Re-Order Buffer(ROB).For this,each ROB entry is appended with a set of bits(equal to the number of PEs),set according to the PE assignment of the consumer instructions. Only the requirement by the original consumer instructions (and not the replicas)is recorded.This information is stored (as a bit-vector called history vector)along with the instruction in the instruction cache.The length of the bit-vector is equal to the,and the information is stored relative to the PE containing the instruction(with the PEs numbered logically).So,a‘1’in thefirst bit of the bit-vector indicates that the instruction’s result is likely to be needed in the same PE,a‘1’in the second bit indicates that the result is likely to be needed in the logical successor PE,and so on. Non-speculative replications are performed for the benefit of the current set of dispatched instructions as in the MyRep technique,and speculative replications are performed based on the history of the instructions4.In this technique,all the instructions falling under the scenario in Figure2(iv)can only be replicated non-speculatively,because no centralized information is maintained regarding the replicas.Figure4gives an example of instruction replication using this technique.In thisfigure,has been already speculatively replicated in PE0,but not in PE1where its result is needed by,because the operand requirement of replicas are not recorded in the history.As in the MyRep technique,and are replicated non-speculatively.When instructions-are in the dispatch stage,it is also recognized that in the near future,the results of and are very likely to be needed in PE3,and the result of in PE2.Among these instructions,and could be replicated speculatively. could be replicated under the scenario in Figure2(iv).On the other hand,is not replicated speculatively because its operand is produced in the same PE.Once these replications are performed,the only delays due to inter-PE communication are from to and from to.D.Implementation DetailsThe replication of instructions is done in the dispatch stage of the pipeline.It is done only if space will still be available in the PEs after dispatching all of the currently fetched(original) instructions.Once dispatched,an instruction is not replicated. Once a replica executes,its result is neither written to the registerfile nor forwarded to other PEs,but is only used within its PE to feed its dependent instructions.To ensure this distinction,an additional bit is used for each instruction entry in the Instruction Window to specify if that entry holds a replica or an original.If the dependent instruction is not able to issue while the result of the replica is in the bypass network,then the dependent instruction has to wait till the result is available from the other clusters.To avoid delays in the execution of original instructions,the originals are given higher priority during issue.However,we also performed experiments where the result of the replica is stored in a small fully associative buffer in each cluster to elongate the period when the replica’s result is available in a cluster.We found that storing the replica’s result for additional cycles improves the performance by a very small amount.Hardware Implementation:Figure5shows a schematic dia-gram of the decode/dispatch stage.For the distribution 4The collection and use of this history is more suitable for base algorithms where the instructions are evenly distributed among the PEs.This is because, in such algorithms the relative distribution of the instructions does not change significantly over time.59i 1345678101112PE 0 PE 1 PE 2 PE 3i i ii i ii i i i i 29i 1345678101112PE 0 PE 1 PE 2 PE 3i i i i i i i ii i i 2(i) Without Replication171816131316Inter−PE communication Intra−PE communicationReplicated Instruction (ii) With Replication10i i 11i 1222i i 24i 2320di1718212320201918212320201918i i i i i i i i i i i i i i i i i i i i Fig.3.A Simplified Example illustrating the Myopic Replication Techniquei i PE 0 PE 1 PE 2 PE 3i i i Speculatively 19ReplicatedInter−PE communication Intra−PE communicationi 6i i i i 12i 3i 9d547i i89i i i 1011122020161317182321181514i i 131i i 2i 39di i i i i i 101112222420d 21d23i i i i i i i i i i Fig.4.A Simplified Example illustrating the LaRep Techniquealgorithm,PE assignment is done in parallel to decode.Also,in parallel to register renaming,the PE assignments of the instructions are compared to determine the instructions that satisfy the replication criteria.For the instructions whose operands are produced by instructionsdispatched in previous cycles,the PE assignments of the producers of their operands are known from the map table.Finally,the dispatcher dis-patches the instructions (with replicas given a lower priority).In this design,we can expect the delay through the dispatch stage to be affected very little by the addition of instruction replication.Nevertheless,we study the impact of additional delays in our experiments.Fig.5.Schematic Figure of the Decode/Dispatch StageInvalidation of Replicas:Once an original instruction exe-cutes,its replicas (if any)need not execute because the result from the original instruction is very likely to be available before the replicas finish their execution.Those replicas can be invalidated,and their entries in the Instruction Window can be recovered.Consider an original instruction that was issued in cycle and finished execution in cycle .For a single-cycle latency crossbar interconnect,its result becomes available in all PEs in cycle .Therefore,all of its replicas that could not issue before cycle can be invalidated.Our experiments showed that almost all of the replicas get ready either in the same cycle as the original instruction or just one cycle before the original instruction,because of the selection criteria used for replicating instructions.Based on this observation,any replica that is ready for cycles but has not yet issued,is invalidated.For this,if the replica is ready at the start of a cycle (i.e.,it became ready in the previous cycle),and is not able to issue in the cycle (i.e.,the replica has been ready for 2cycles),it is invalidated.Also,if a replica executes and forwards its result to the waiting instructions,it is immediately invalidated and its entry in the Instruction Window recovered.This helps in reducing the space occupied by the replicas.History Collection Techniques:For the LaRep scheme,only the last PE assignment of an instruction’s consumer instructions are noted to determine the speculative replication6of that instruction the next time it is dispatched.We call this technique of collecting history as last assignment.The last as-signment technique may suffer from unnecessary replications if the assignment of the dependent instructions changes.This limitation is only for the speculatively replicated instructions. We stick to the last assignment technique for its simplicity, as it permits the history of the instruction to be easily stored along with the instruction.Also,our experiments showed that with more complex history collection techniques,the increase in IPC is negligible.Our experiments also showed that the speculative replication accuracy increases by only about10% when using a pattern-based5history collection technique as compared to the last assignment technique.Replicating Load Instructions:Load instructions load from memory data values,which may be required in multiple PEs.A simple replication of a load instruction will result in multiple accesses to the cache to load the same value.An alternative to replicating the load instruction is to make the cache broadcast the load value to all the PEs where the value is needed.When a load instruction is about to be dispatched,the PE numbers where its value is required are stored along with the load instruction.Eventually,when the load instruction is issued,the read request to the cache is accompanied by the PE numbers where the result is needed.Branch Misprediction:In our experiments,all the instruc-tions(the originals as well as the replicas)after a mispredicted branch are squashed.Interconnect Bandwidth Requirements:For the crossbar interconnect used in our clustered processor configuration,we assume that all the results produced in a PE are available in all the other PEs after a delay equal to the inter-PE communication latency for the crossbar interconnect.We also assume that all the results produced in a PE are broadcasted to all other PEs for waking up the dependent instructions, as is done in Alpha21264[16].When using instruction replication,the bandwidth requirements and the number of values communicated across the interconnect do not change (note that the replicas do not broadcast their results).IV.E XPERIMENTAL E VALUATIONA.Experimental SetupFor our experiments,we modified the SimpleScalar[7] simulator to simulate clustered processors.Our simulator simulates a64-bit PISA architecture.The default parameters are given in Table2.For benchmarks,we use a collection of8INT(bzip2,gzip,parser,twolf,vpr,gcc, crafty and vortex)and2FP(equake,and mesa) programs from the SPEC2000suite,compiled with an-O3flag.We also experiment with a high IPC integer matrix multiplication benchmark.The SPEC2000FP bench-marks(among the ones written in C Language)and the SPEC2000integer benchmarks chosen are those that have a significant IPC reduction when run on clustered processors. 5In this technique,a number of previous controlflow patterns are recorded for each instruction.For each pattern,confidence counters are used to determine the specific PEs where the instruction is to be replicated.Performance statistics are collected for500M instructions after skipping thefirst500M instructions.All of the techniques use[5]as the base load balancing-oriented distribution algo-rithm.We also use a single-cycle latency crossbar interconnect.B.Experimental Results and AnalysisFigure6gives the percentage increase in IPC for various configurations of clustered processors compared to the IPC obtained with the original distribution algorithm(which were already presented in Table I).Figure6gives the IPCs for both a single-stage dispatch as well as a2-stage dispatch (for the case where instruction replication incurs an addi-tional pipeline stage).As seen in Figure6,when compared to original distribution,the IPC obtained from increases by about5-10%with the MyRep scheme and by about5-12%with the LaRep scheme.The IPCs with a2-stage dispatch are slightly lower than a single-stage dispatch because of an increase in branch misprediction penalty.More importantly,when compared to the original dependence-based algorithm(cf.Section2),with instruction replication is performing better for many of the benchmarks,indicating that the approach of instruction replication on a load balancing-oriented base instruction distribution algorithm may even be better than using an algorithm giving importance to both inter-PE communication and load balancing.Figure6indicates that this is may be true even when instruction replication adds an additional pipeline stage.As expected,instruction replication does not improve the performance of dependence-based algo-rithm significantly,because there are not that many instructions (as in)suffering from inter-PE communication delays. Next we present an in-depth analysis of the results obtained for a single-stage dispatch configuration;similar results were observed for a2-stage dispatch stage,but these statistics are not reported to conserve space.For this analysis,we first measured the percentage of instructions(out of the total dispatched)that suffer a delay due to inter-PE communication. These results are shown in Figure7(a).Thisfigure throws more light on the IPC results presented in Figure6.Figure 7(a)explains why MyRep and LaRep schemes perform almost the same,with the LaRep scheme performing slightly better. When comparing these statistics with the original dependence-based algorithm,it is seen that the number of instructions delayed in dependence-based algorithm is less than the number of instructions delayed in with instruction replication. Still,the dependence-based algorithm performs worse than instruction replication for most of the benchmarks in Figure6. This is because the load imbalance incurred by dependence-based algorithm is much higher than.We also measured the total number of instructions repli-cated,the total number of replicas that could issue,and the total number of instruction replication attempts that did not go through due to lack of space in the instruction windows. Figure7(b)gives these results in a stacked bar for the MyRep and LaRep techniques using as the base distribution, and for the LaRep technique using dependence-based as the base algorithm.These statistics do not include the replicated load instructions(as there is no physical replication for them).。