White Paper_InterPhonic

第1章 Chapter 10 Company Introduction Anhui USTC iFlyTek Information Science & Technology Corporation Limited is a national leading software company specialized in Intellegent Speech and Language Technology research, software and IC product development, Speech Information Service and Electronic Government System Integration. For the numerous software companies in China, USTC iFlyTek is one of the few companies that has the core technology and holds the self-determination intellectual properties. His core intelligent speech technology represents the highest level in the world. In 2008, USTC iFlyTek is listed on board of Shenzhen Stock Exchange. The stock share code is 002230.USTC iFlyTek was founded in 1999 with the registered capital of 80.36 million Yuan. The main stock share holders include: the Sci&Tech Industrial Company of USTC, Shanghai Guangxin Corp., Legend Investment Inc. and Infotec Corp.As a national industrialized entity, our company has the longest basic research history, the largest capital scale, the best results in the previous evaluations, the most specialized talents and the highest market share rates in the area of Speech Technology.The speech tech realized the Human-Machine interaction to make the Human-Machine communication as easy as Human-Human communication. Speech Technologies mainly include the two key technologies of Speech Synthesis and Speech Recognition. The Speech synthesis technology is used to make the machine speaking; while the speech recognition technology is used to make the machine understanding human’s speech. In addition, the Speech technologies include Speech Coding, Tone-color Conversion, Pronunciation Evaluation, Speech Denoise, Speech Enhancement and other technologies, having extensive application space.As the largest vendor of Intellegent Speech Technology in China, the USTC iFlyTek corporation had experienced a long-term research on the intelligent speech technologies. And we have made a lot of international leading achievements in Chinese Speech Synthesis, Speech Recognition, Pronunciation Evaluation and other technology areas.USTC iFlyTek is the only “Achievement Industrialization Base of the National 863 Program” in speech technology area, the key Software Enterprise under the National layout, the high-tech enterprise under the national torch program, and one of the National High-tech Industrialization Demonstration Projects.Our company has been designated as the leader unit for the Chinese Speech Interactive Technology Standard Group by the Information Industry Department to lead the development of the Chinese Speech Technology Standard. In 2003, USTCiFlyTek was awarded the only “National Science & Technology Progress Award (second class)” in the speech industry. In 2005, the company was awarded the “Significant Technology Invention Award in Information Industry”, which is the highest honor for the independent innovation in Chinese Information Industry. In 2006, 2007 and 2008, we continued to hold the title of the first place in three successive International Blizzard Challenges of English Speech Synthesis.Based on the world leading intelligent speech technology with the self-determination intellectual properties, USTC iFlyTek has a lot of products for various applications, ranging from large-scale telecommunication application to small-scale embedded applications, from the telecommunication and finance industries to the small enterprise and family customers, and from PCs and mobiles to MP3/MP4/PMP and toys. USTC iFlyTek holds more than 60 percent share rate in Chinese Speech Technology market, more than 70 percent share rate in Speech synthesis product market, more than 80 percent share rate in telecommunication, finance, electric power, social insurance and other mainstream industries. We have more than 500 development parteners. The Chinese Speech Industry based on iFly technology has accquired a considerable scale.USTC iFlyTek’s outstanding achievements in the speech technology research and industrialization have attracted extensive attention from the social societies. Wu Bangguo, Jia Qinglin, Li Changchun, Xi Jingping, He Guoqiang and many other national leaders and party leaders has inspected our company. They all highly appraised the innovative efforts made by USTC iFlyTek. The national administrations and authorities had formed a common understanding that USTC iFlyTek is the “national team” in Chinese Speech Industry.Company HonoursThe Achievement Industrialization Base in National 863 ProgramThe Key Software Enterprise under the National LayoutThe Key High-tech Enterprise in the National Torch ProgramThe National High-tech Industrialization Demonstration Project BaseThe National-level Outstanding Software EnterpriseThe 100 Best Enterprise in Chinse Electronic Government Affair IT IndustryThe leader unit of the Chinese Speech Interactive Technology Standard workgroupThe national post-doc R&D workstation第2章 Chapter 11 Product Introduction2.1System DescriptionSpeech Synthesis is the technology to convert the text into speech data. It involves the text analysis & processing, digital signal processing and various cutting edge technologies. InterPhonic Speech Synthesis System is the independently developed Chinese-English Mixture Speech Synthesis system by Anhui USTC iFlyTek Information Technology Corporation Limited. It mainly aims at the industry-level speech information service, and provides the services by real-time and dynamic text conversion to fluent and natural speech data with high quality.Text-to-Speech is also known as Text-Speech conversion. It involves acoustics, linguistics, digital signal processing, multimedia and other disciplines. It is one of the cutting edge technologies in Chinese Information Processing area.As a top info-tech research area, the development of the speech synthesis technology, especially the Chinese speech synthesis technology, has been always highly concerned by the industry.Under the support of the National 863 Program, the National Key Technology R&D Program during the 9th Five-year Plan period, the NSFC project and the Key Program in CAS’s 8th Five-year plan, the speech synthesis technology of USTC iFlyTek has experienced more than 10 years’ rapid growth, and attained the leading position in the speech synthesis technology area. In order to keep the leading position worldwide, USTC iFlyTek company has built many joint labs with the CAS Acoustic Institute, the Linguistic Institute of Chinese Academy of Social Science and other domestic outstanding institutes. We have also built the stragetic cooperations with Lenovo, Huawei and other leading companies, laying the solid foundation for the further development of the speech technologies. In 2006 and 2007, we continued to hold the title of the first place in two successive International Blizzard Challenges of English Speech Synthesis.InterPhonic speech synthesis system is based on the Large-scale speech material library and the Trainalble TTS technology, and can produce natural, fluent speech comparable to human’s voice. It can continuously synthesize the arbitrary text. InterPhonic Speech Synthesis System attemps to set up and improve the Human-Machine speech interface, and to provide efficient and stable speech synthesis performance for the large-scale speech service application. It can provide an overall solution for the Telecommunication, Enterprise-level and Desktop-level applications. It becomes the intrinsic power for the novel speech services, speech websites and multi-media office work & education.For the customers who want to know the detailed functions and features of the InterPhonic Speech Synthesis System, this document provides them all details on InterPhonic product specifications. The reader will get to know the overall functions of the world-leading speech synthesis system and understand the important concepts and features of the synthesis system.The schematic for the Speech Synthesis Technology is shown below:From the above figure, we can see that the synthesis engine of the system takes two simple steps to convert text into speech data: The text will firstly be processed in the front-end grammar analysis. Through the dedicated dictionary and rules carefully developed by USTC iFlyTek, the text will be converted into standard format with grammar-level information, and then be transferred to the rear end. The rear-end will take rhythm analyzing and processing based on the result of the front-end analysis to get the speech’s time-duration, prosody and other rhythm information, and then pick the most suitable speech unit in the library according to the obtained rhythm information. The speech unit will be further processed and concatenated to form the final speech output.The overall processing involves a lot of Chinese-English grammar analysis, rhythm knowledges, grammar & semantic analysis algorithms, optimization path search, unit selection and adjustment algorithms, and the knowledge on speech data coding as well. The speech synthesis technology covers not only the social sciences of speech and literatue, but also the natural science like digital signal processing. It’s a typical multi-disciplinary high-tech.The Speech Synthesis product has a great dependence on the core technology. In order to keep long-term advantages in core technology and to ensure transferring the core tech to the high quality products with satisfactory functions, stabilities and easy-to-use features, USTC iFlyTek takes a unique industry and research combined mode to improve the research level and product quality simultaneously. The company cooperates with CAS Acoustic Institute, USTC and CASS Linguistic Institute to buildthe joint labs, and complete some forward-looking research in these labs. Such alliance with first-class parterners and advantageous complementary ensure the continuous improvement in core tech and the leading position of iFlyTek in speech technology area. Through the independent R&D, USTC iFlyTek has acquired many technical achievements with originality in various areas of Chinese Speech synthesis technology.The unique Speech Material Statistical ModelThe Speech Material Library Design Approach with consistent fore end and rear end, and the automatic Speech Material Library Creation approach The high-accuracy Rhythm model based on length-varing rhythm templates and hearing quantificationRobust intelligent text analysis & processing technologySpeech Material Library cutting technology with the minimum hearing loss The multiple-language speech synthesis framework with separated language knowledge and system modelingThe customized speech syntheis technology for particular applicationsHMM-based Wave Concatenation Technology2.2Speech Library ManagementInterPhonic Speech Synthesis System allows to install multiple speech libraries with different voicers and different sound sample rates. The currently supported speech libraries are listed below:SpeechLibrary Number Voicer CharacteristicSupportingLanguagesSupportedSample RateYoung lady’svoiceChinese andChinese-EnglishmixtureYoung lady’svoiceCantonese-EnglishmixtureMiddle-ageman’s voiceChinese-EnglishmixtureYoung lady’svoiceChinese-English mixtureYoung lady’svoiceTaiwan-style ChineseMiddle-agelady’s voicePure MandarinMiddle-ageman’s voiceAmerican EnglishYoung lady’svoice American EnglishThe user can manage the present installed speech libraries in the “Resource Management Center” to support the Adding new library, Deleting library, Loading/unloading library and default library functions.Select the speech library using method for the synthesis:1. Open the “Resource Management Center” program and change the default speechlibrary option in the “speech library management” directory.2. During the development process through iFly TTS SDK, designate the relevantparameters of the relevant interface functions to finish the setting.3. Select the speech library directly in the main page of the demo program.Technical SpecificationsItemsSpecificationsSupported OSSupported LanguagesMandarin, English, Cantonese Supported Speech LibrariesChinese & English of Lady’s voice, Chinese & English of gentleman’s voice, pure English library Supported text control marks SupportCSSML and SSML text marks Supportedspeech data format Sample rate Speech data formatSpeed adjustmentChinese-English mixtureEnglish-string processingPronounced according to the English custom Supported character setRecognized Chinese Character number Support GBK character library of 21003 traditional & simplified chinese charactersThe phrase number in the system vocaburaryUser-defined vocaburarySpeech Data Buffer settingsSentence separation parameter settings Support the carriage return parameter and the maximum sentence-separation length parameterPunctuation pronunciation settings Can choose the pronunciation method for the punctuations and the detailed pronunciation locationSpecial symbols’ pronunciation settingsSupport special symbols’ pronunciation settings Digit-string’s pronunciation settingsSupport choosing the pronunciation of digit string Pause style settingsSupport choosing the pause style Rhythm correctness rateUnderstandability of the sentenceNatural degree 4.3 (5 for maximum)2.3 ApplicationThe speech synthesis technology can automatically transfer the arbitrary text info to continuous speech in real time. It’s an efficient and convenient tool for providing speech services for any person, at any time and any location. This technology perfectly meets the requirements of huge-amount data, dynamic refresh and customized inquiry in this information era. InterPhonic speech synthesis system can provide efficient and flexible services for applications in various areas, such as:z PC speech interactive entertainment and educationz Telecommunication level and enterprise level call center platformz UMS and voice portal speech service systemsApplications of InterPhonic facilitate the change of the original speech information production method by replacing the traditional simple concatenating, human recording and operator services. In large scale speech application system, the multi-channel real-time speech synthesis service will not only improve the time-effectiveness of the information updates but also provide significantly extended speech information scope. The high-quality speech effect, flexible application andproven stability of the InterPhonic speech synthesis system have enabled many customers to replace the traditional operator service by the self-assistant speech service. With the assured application effects, it has provided the services with higher automation degree, lower cost and higher quality.2.4Documents and relevant materialsYou can refer to the following documents and materials for the system development and usage:The InterPhonic development manual provided along with the productInterPhonic Help center, the online help document, which is located in the help folder of the installation directory.The development interface example programs, located in the TTS SDK folder of the installation directory.2.5Customization and Optimization2.5.1Custom Resource PackageIn the practical applications, the synthesis text sometimes belongs to special areas (bank area for example) and has obvious feature in content and form. USTC iFlyTek proposed the area customization concept for this situation. Area Customization is to use the custom resource package to improve the speech effects in specific area’s application. The “Custom Resource Package” is an assembly of resources which will be used by the synthesis engine in special area applications, including the speech library, vocaburary, external rule library and user-defined symble library. Using the custom resource package can help improve the speech synthesis effects in the special area.2.5.2CSSML Text Mark LanguageSSML (Speech Synthesis Mark Language) is the standard synthesis text mark language developed by the W3C international standardization organization. The SSML uses XML to define the various marks for describing the synthesis text processing. These marks can be roughly classified into three groups: grammar level marks (such as paragraphs, sentences and words), marks for defining grammar characteristics (such as text pronunciation), and marks for defining the rhythm features (such as speech speed, prosody and volume). But the SSML is mainly developed for English marks, and can not meet the unique requirements of Chinese Speech Synthesis applications.CSSML （Chinese Speech Synthesis Mark Language）is a Chinese Speech Data Description Specification proposed and primarily developed by USTC iFlyTek. This standard obtained great concern and support from the National 863 Expert group, the national information standard committee and the national quality & technology supervision bureau. Now the national Information Industry Administration has signed off the official document to establish the dedicated workgroup to promote the devopment of this standard.CSSML is based on VoiceXML, and takes the characteristic of Chinese synthesis into consideration. It expanded the SSML marks as necessary to make them more suitable for Chinese speech interactive application. Meanwhile it keeps full compatibility with the SSML 1.0 specification. Currently the CSSML can make flexible marks on the phonetic symbols, stressed/unstressed pronunciation, rhythm layers, prosody emphasis and background music. It also has good expansibility which enables the effective improvement on the practical capability of the Speech Synthesis system.Through the CSSML parameter settings, you can specify the specific synthesis parameters for the text (unlike the “resource management tool” provided by the synthesis engine that the settings take effects on all texts), to implement more flexible control on the synthesis effects.InterPhonic system can automatically recognize and process the CSSML text with standard format, and properly support the individual marks.The user can designate the text as CSSML format in calling the relevant interface functions. The visualized edit for the text to be synthesized can be realized in the CSSML Editor.2.6ToolsInterPhonic provides many speech application tools, including:Synthesis Demo ProgramThe synthesis demo program (ttscon) uses the synthesis engine interface libiflytts.so for synthesis. It is used to demonstrate the effects of the speech synthesis. The user can input a section of text and listen to the synthesized speech, or he can adjust the frequently used parameters to test their impact on the synthesis effects. Through adjusting the parameters and exporting the speech files, the user can experience the basic functions of the system. The demo program can also be used to verify the system installation.Encryption Lock inspection programThe encryption lock is used to prevent illegal copy and control the license number.The inspection program will verify the product name, version, license number and license expire date of the speech synthesis syste.TTSDoctorTTS Doctor is an automatic technical support tool for the Speech Synthesis System software developed by USTC iFly corporation. This tool will be installed onto user’s machine along with the InterPhonic Speech Synthesis System.The main functions of the TTSDoctor include: checking the installed TTS product information, viewing the system information, assisting error diagnosis and obtaining technical support.User can use TTS Doctor to collect the information about the local machine and save it in “.ttd” file as the maintenance reference. When the user encounters any problem during InterPhonic Speech Synthesis System usage, he can send this file to our technical support personnel to obtain quick and effective technical support.CEditorCSSML Editor is a visualized text editor for CSSML text mark language that is particularly developed by USTC iFly corporation.Through CSSML Editor, you can use the CSSML marks to correct the text with bad synthesis effects, to check whether the CSSML grammar is standard and to convert the text format.Resource Management CenterThe resource management center program is the management center for the InterPhonic Speech synthesis system. It governs all the parameters and resources which could affect the system and the synthesis. The settings in the Resource Management Center will affect the registry, resource files and other global resources. If they are modified, the updated items will take effects after the next system startup.第3章 Chapter 12 System Performance3.1Performance MeasurementThe main function of the system is to convert the text into the speech data. The quickness of this conversion process will affect the performance of the system applications in providing speech services. If the conversion is too slow, some problems maybe happen such as incontinuous speech play and long waiting time. So the system performance characteristic is also a key specification related to the system application effect.For multi-channel speech synthesis system, the most obvious way to measure the performance is to count the time required for synthesizing a certain amount of text for each channel. Less time needed implies better system performance. In the long-term cooperation with the integration and operation vendors of telephone speech application, USTC iFlyTek gets to know that in the speech service business the best synthesis response time is about 3 seconds and can not exceed 6 seconds, otherwise the user will feel that the waiting time is too long. The synthesized speech play is normally based on sentences, and the text length should be within 50 Chinese characters (100 bytes). For this reason, USTC iFlyTek uses “Synthesis Time” to indicate the quickness in the performance test. This term is defined to the average time for synthesizing 50 Chinese characters (100 bytes). According to the practical requirements in Telephone Speech Applications, the “real-time” characteristic in the system performance is defined as: In case of multi-channel parallel text synthesis, if each channel can finish the synthesis within 6 seconds and the average time is about 3 second, the system can be deemed to be able to satisify the real-time requirement in the parallel application with this channel number.In order to accurately measure the system performance, USTC iFlyTek used the home-made MTTester tool to make the uninterrupted, high-density and multi-channel synthesis service tests, and took the data obtained from the above test as the performance data. The text tool uses different text for each channel, and gets the total time for synthesizing all the text files in simultaneous situation. The equivalent time for 50 Chinese characters is the “Synthesis Time”. As increasing the channel number, the system CPU and memory consumption will be greater and greater, and the “Synthesis time” for each channel will also gradually increase. When the Synthesis Time reaches the critical condition of real time requirement, the channel number is the “maximum parallel synthesis channel number” in this test environment. The “maximum parallel synthesis channel number” also indicates the system performance, with the higher value for higher performance.The overall performance of the InterPhonic system is counted by throughput, which is dependent on the license number. According to the above description, if the average synthesis efficiency is 2 seconds for every 50 Chinese characters (25 Chinese characters a second), then the overall synthesis throughput of the system is 25 Chinese characters a second multipled by License Number.3.2Facts affecting the performanceIn order to achieve the best synthesis effect, the InterPhonic speech synthesis product used very complicated algorithms and a lot of resource files (such as speech libraries and vocaburaries). So the system performance specification is affected by many software and hardware aspects. Next we will analyze and describe these aspects.1.The algorithm complexibility of the speech synthesis system itself: Because the internal algorithm and resource for each speech synthesis system edition is different, the performance for each edition is not in the same level. Sometimes the performance difference between editions is huge.2.The calculation ability of the Test Server CPU: Because a lot of logic and mathematic calculations are to run inside the synthesis system, the CPU calculation ability (expressed by the CPU frequency) and CPU number will greatly affect the performance of the system. More CPUs and higher frequency will result in better performance.3.Memory: during the operation of the speech synthesis system, a great amount of intermediate data is to be saved in the memory, and these data will experience frequent operations and calculations. So the memory size and the access speed have significant impact on the system performance. If the memory volume is smaller than the required amount, the system will not present the best performance. With faster memory access, the system will get better performance. We recommend that the InterPhonic runs in the computer with 2G or more memory.4.Operating System: Each kind of OS has different stragety in process/thread calling, memory management and network service. According to the test results of USTC iFlyTek, the Server Edition products (such as Advance Server 3.0) have better performance and higher stability than the Desktop Edition products (such as Redhat9.0). So we recommend to use the server edition in the large scale speech synthesis service applications.5.Hard Disk: Because all the resource files in the synthesis system have largesizes up to Gbytes, and the system requires frequent data exchange with the hard disk during synthesis process (especially in the service startup period), the hard disk random access speed will affect the synthesis performance. We recommend the customer to use the IDE HD of more than 7200 rpm speed or the high speed SCSI HD on the speech synthesis server host.work Environment: In the application of remote service calling via network, the network configuration will also greatly affect the system performance. In a parallel Speech Synthesis Service host with more than 100 channels, the network exchanged data amount will reach tens of Mbps or even more. So we recommend to use the network card with more than 100 Mbps bandwidth on the client and server ends, and to place the two ends in the same Local Area Network. And we strongly recommend to use the 100Mbps Network Exchange machine but not 100Mbps hub. If the LAN bandwidth is 10Mbps, the network service efficiency of the speech synthesis will be significantly limited.3.3Test Environment3.3.1Standard Hardware Test EnvironmentServer End: P4 Xeon 3.0G Dual core and dual CPU, 2G memory, 140G SCSI hard disk and 100M Ethernet network cardClient End: P4 2.4G CPU, 512M memory and 100M Ethernet Network cardDistributive Client End: P4 3.0G CPU, 1G memory, 80G Hard Disk and 100M Ethernet Network card3.3.2Standard Software Test EnvironmentiSP Network EnvironmentWindows2000/2003/XP/VISTAInterPhonic Speech Synthesis system (Enterprise Edition, with 8K speech library)3.3.3Basic Environment“Standard Hardware test Environment”, the hardware environment for the local test & use in the server end;“Standard Software test Environment”, the software environment for the local test & use in the server end;。