cpsc462project

Open Project: Basic Augmented RealityChad LoefflerMarcus MojicaCPSC 462 – Spring 2004March 31, 2004 - May 13, 2004Table of ContentsTable of Contents 2Introduction 3System Design 3Implementation Detail 6How Our Project Can Be Improved 8Major Files 9Materials 10Figures 11References 12Connection 13Code AttachedIntroductionAt the heart of our open project are the concepts that may one day change our way of viewing and interacting with the world. In this project, we explore the building blocks for augmented reality using infrared, a USB camera, eb63, and FPGA. One of our extra objectives was to add audio to the project.Augmented reality is the concept of taking virtual objects and overlaying them in real world space. A virtual object is the representation of a real object. Even though emphasis is placed on visual presentation, virtual objects may include sound, movement, interaction, and so forth.System DesignOverviewThe system design consists of a 4-phase augmented reality system based on the proximity lookup model. This model makes use of two types of devices, a server and a client. When the two devices come into the range and the line of sight of each other, a virtual object id is transferred. The virtual object is retrieved and displayed.Phase 1 established a working detection system using infrared. Simply, when the correct infrared signal is detected and it is in the line of sight, the state of the system changes. A client device (FPGA) listens, detects a state change, and responds by sending a print to screen.Phase 2 continuously broadcasts an id using infrared from a server device (eb63, FPGA, and breadboard circuits) and transfers it correctly to the client. The client looks up the associated id and displays the associated text or image. As long as the server deviceand client device are in the line of sight, the text or image is displayed. Phase 3 extends Phase 2 by finding a technique to overlay an image on real-time video (see Figure 1).Phrase 4 was not originally included in the project proposal, but was added since phrase 3 was already reached earlier than expected. The breadboard circuits were transferred from the breadboard to PC boards and the static virtual objects were placed in a project box. Speech recognition was added to allow the eb63 to be a dynamic virtual object.Project BoxThe project box is simple in design. Inside, the project box contains two circuits that act independent of each other, but share a common power supply. One operates a infrared emitter at a certain hertz while another infrared emitter operate another specified hertz. The hertz act as the virtual object id.EB63We expanded the project by adding speech recognition. Microsoft Mary is the voice that comes with the Microsoft Speech SDK. When we receive the blink data, Mary tells us what the blink rate is, what the average blink rate is, and what animal corresponds to that blink rate. For the other end of our project, Mary also takes commands from us through the Coffee Shop interface. When we give a command that she recognizes, she takes the corresponding action. These actions involve making the eb63 blink at a certain rate per time interval.For the eb63 side of the project, here was the high-level design. A program called “Coffee Shop 6” runs. It is a modified version of one of the example programs supplied with the Speech SDK. The program listens for a command from us. If we give it acommand that it recognizes, it will take action. The action is to find which number corresponds to the animal name that was supplied with the command and write that number to a file. Another program is continuously reading the number from that file and outputting it to the parallel port. The eb63, which is polling the parallel port, reads in the number and finds what animal that number corresponds to. It will then blink at a rate specified by us to emulate the transmitter that corresponds to that animal.FPGAFor the FPGA side, which is on a second computer, here is the high level design. The FPGA is using the signal from the receiver as a clock source to a counter. This counter is reset at the beginning of each time interval. Before reset, the counter output is latched into a register. This register output will be the final counter value for the previous time interval. This data is sent to the parallel port, where a program reads it as the blink rate. The program then speaks the blink rate to us. After getting 3 good blink rates, it will calculate the average blink rate and tell it to us. It will also tell us what animal corresponds to that blink rate. The program will also create a file with a picture of that animal. This picture file is read by a JavaScript program in a web browser and overlaid on a web cam image.One major note is that the Microsoft Speech Engine requires Windows libraries that come as part of Microsoft compilers. We could only compile our code using Visual C++, Visual Studio, or Visual .NET C++. While the documentation says the speech engine will work with any 32 bit C++ compiler, it does state that any example code given requires a Visual product.Implementation DetailProject BoxThe circuits for the project box were basic circuits. Figure 1 contains each circuit specification. The circuit contained a 555 timer configured for astable multivibrator mode with an infrared emitter connected to the output. Even though the circuits were basic, this part of the project took about 4 weeks to configure correctly. This was due to bad and incorrect 555 timers. Once this was fixed, however, the project moved along quickly. The circuits were configured for 1 and 20 hertz. The 20 hertz was obtained by modifying the capacitor value in the figure specifications.EB63Here are the details for the computer with the eb63. The Coffee Shop 6 programis listening for a command. Because of limited time, we were unable to create our own Grammar. A Grammar is a set of speech rules and commands. We had to use the existing Grammar and append our commands to the preexisting commands.The section of the program that renames the coffee shop was ideal for our purpose. It uses a preexisting command of “Rename the Coffee Shop to <new name>”, where<new name> is anything that you say. In order to modify this to work the way we wanted, we added an intermediate step in its processing function so that we could find what the new name was. We then used our animal names as the new shop name. We have 4 commands: “Rename the Coffee Shop to dog”, “Rename the Coffee Shop to cat”, “Rename the Coffee Shop to mockingbird”, and “Rename the Coffee Shop to stop”. The first 3 commands are animal names. The last command is to make the eb63 stop blinking.We found that the recognition algorithm could correctly identify our commands most of the time, but not every time. This caused us to repeat commands, which annoyed others in the lab. As a quick workaround, we modified our command processing function to check for the first letter only. Thus, our commands of “dog”, “cat”, “mockingbird”, and “stop” could be replaced with “dollar”, “chicken”, “marshmallow”, and “sushi” for example.Since we could not get the parallel port code to run with the GUI based program, we had to have the program output to a file. We then used a command line program to read from the file and output that to the parallel port. The eb63 was using a modified version of the LCD program to poll for data. After receiving the data, it would adjust the number of milliseconds in its wait function. The eb63 was running an infinite while loop and used a wait function to pause at a certain time interval. At each new interval, its transmitter output would toggle. By having it wait the right amount, we could adjust the output blinking frequency of the transmitter.FPGAA second computer was used for the FPGA side. The FPGA had the receiver attached to one of its I/O pins. The FPGA used the signal from the receiver as a clock source to a counter. This counter was reset at the beginning of each time interval. Before being reset, the counter output would be latched into a register. This register output would contain the final counter value for the previous time interval.This data was then transferred to the parallel port, where a program read it as the blink rate. The blink rate would then be spoken by the speech engine. If 3 consecutiveblink rates were all within 20 of each other, then their average would be calculated. This average would be spoken along with the name of the animal that it represented.Before the average was spoken, a picture file was modified to contain the pictureof the specified animal. This picture was then overlaid upon a semi-live web cam image. The Microsoft Speech SDK 5.1 supplied the text-to-speech engine along with the voiceof Microsoft Mary and the speech recognition engine.How Our Project Can Be ImprovedThe project could be improved in a number of ways. True real-time video and overlay could be handled by a single C++ program instead of going through web browser software and simply updating still images. A real virtual object ID could be transferred and compared to a table instead of simply deducing it from the hertz. Instead of infrared, RFid technology would be a much more practical for previous suggestions.As for speech recognition, creating the Grammar Context for the speech recognition engine and creating a GUI-based program around this. Instead of using preexisting commands and appending their commands, they could create their own commands and parameters.ConclusionThe project was an excellent learning opportunity. We were able to achieve our goals (and extend them) in the given time period. This is augmented reality on small scale.Major FilesProject_top.vMain FPGA fileCounter.v8-bit counter with asynchronous reset generated by CoregenIPLCDObj.ceb63 fileCoffee.cppCoffee Shop Program using the speech recognition engineInpoutTest.cppreads a number from a file and outputs it to the parallel portProject.cppreads blink rate, speaks using speech engine, and changes picture filecam.jpgThis image is replaced continuously by web cam software such as ConquerCam. The highest possible interval is suggested to simulate a real-time video feed.overlay.jpgThis image is replaced by the project's parallel port program. When the identification of an object is discovered by the FPGA, the parallel port program determines the associated overlay image. The overlay image is updated.changer.batThis program simulates the process of updating cam.jpg and overlay.jpg. It simply swaps the current images. Note: this will only work by copying the files listed here to hard disk. start.htmlThis HTML document contains the introductory.view.htmlThis HTML document brings the project together. The viewer makes use of JavaScript and Cascading Styles Sheets. The script refreshes the images, and the style sheets position the images. The page is opened in a javascript popup window from the "start.html" page for a better visual appearance. This helps to remove toolbars, remove scrollbars, set height, etc.MaterialsFiguresFigure 1 (Source: Tokheim)Figure 2ReferencesLing, Junyi. C++ Parallel Port Code.Tokheim, Roger L. Schaum’s Outline of Theory and Problems of Digital Principles. 3rd ed. New York: McGraw-Hill, 1994.ConnectionTo connect the eb63 to the parallel port, a cable is needed that converts the parallel interface to a small-pin interface. The parallel port D0 and D1 pins are connected to the first 2 pins of the PioA controller. These are TCLK3 and TIOA3. These are the 10th and 11th pins on column B of the P3 section of the eb63. A ground pin of the eb63 must be connected to a ground pin of the parallel port for them to work correctly. The transmitter is connected to the PB8 pin and a ground pin. The PB8 pin is the 17th pin on the A column of the P3 area.The FPGA had the receiver connected to pin 123. The receiver was connected only to this pin. The receiver circuit consisted of 2 resistors in series with the receiver. One part of the receiver was connected to the positive power lead. The wire connecting to the FPGA was connected to the part of the receiver facing the resistors. The receiver circuit operated with 4.5 volts. The resistors were a voltage divider. The transmitter of the FPGA was connected to Pin 120 and a ground pin. The project box with the transmitters was operating at 3 V.。