T-86.161 ICTEC Project course – B2B integration XML Schema Based Invoice Validator

T-86.161 ICTEC Project course – B2B integration XML Schema Based Invoice ValidatorMikko Fallenius, Karri Karanko, Samuli ViitanenMikko.Fallenius@iki.fi, karkar@iki.fi, iviitane@cc.hut.fiAbstractWith an increasing pace companies are moving into electronic invoices.Currently, 200 million invoices are sent from one company to another in Finland. Only approximately four percent of these are in an electronic format.This amount will increase rapidly in the following years. This trend is also leading into the development of multiple standards of electronic invoicing. These standards in general are not compatible with each other, and the wide range of business accounting software only makes it more complicated as their support for these standards is very sporadic.Traditionally the solution for electronic commerce has been EDI, but this has been a viable solution only for large, well-established companies.As more and more small and medium-sized enterprises are also moving into electronic commerce, it is important to provide more light-weight, flexible solutions for electronic commerce.In order for electronic invoicing to function well from one company to another, using any of the existing invoicing operators, there has to be a possibity to seamlessly transform these invoice documents from one format to another. This research was conducted to study the possibilities of building an online service for validating XML based invoice documents, which could also produce user-friendly error logs of validation the validation process. This kind of service would be useful in testing electronic invoice documents.Table of Contents1 Definitions (3)2 Introduction (4)2.1 Background (4)2.2 Research Problem (4)2.3 Objectives (5)2.4 Scope (5)2.5 Methodology (6)2.6 Structure of the report (7)3 Existing work (7)3.1 Electronic invoice standards (7)3.2 TIEKE's eInvoice Forum (7)3.3 XML Schema technologies (8)3.3.1 XSD (XML Schema Definition) (8)3.3.2 RELAX NG (8)3.3.3 Schematron (9)3.4 XML Schema based validators (10)3.4.1 Existing online validation services (10)3.4.2 Existing useful software packages (10)3.5 Invoice validators (11)4 Validator service (11)4.1 Requirements (11)Validating an invoice (12)4.2 Functionality (13)4.3 Architecture (15)4.4 Technical details (16)4.4.1 Server requirements (16)4.4.2 Client requirements (17)4.5 Testing the simple validator (17)4.5.1 Test cases (18)Validating against a valid document (18)Validating against a not-well-formed document (18)Validating against an element with false simple type (18)Validating against an element that is written in capital letters (19)Validating against a document with false sequence order (19)4.5.2 Conclusions (19)4.6 Implementation of the XML validator (20)4.7 Discussion about the validator (20)5 Conclusions (22)5.1 Future development (23)6 References (24)7 Appendices (25)1 DefinitionsAPI Application Programming Interface describes ways inwhich pieces of computer software function with oneanother.B2B Invoice B2B (Business-To-Business) invoice means an invoicethat is sent from one company to another.B2C Invoice B2C (Business-To-Consumer) invoice means an invoicethat is sent from a company to its client (or other type ofstakeholder).DTD Document Type Definition. DTD can be used to describethe structure of XML documents.EDI Electronic Data Interchange. A standard for electroniccommerce, originally developed in the 1970s.UN/EDIFACT has been promoting EDI standards inEurope.eInvoice A Nordic standard for electronic invoicing, developed bythe eInvoice Consortium.Finvoice A standard for electronic invoicing developed by theFinnish Bankers' Association.OASIS Organization for the Advancement of StructuredInformation Standards. OASIS is a non-profit, globalconsortium that drives the development, convergence andadoption of e-business standards.PostiXML A standard for electronic invoicing developed by FinlandPost Group.Schematron A language for making assertions about patterns in XMLdocuments.TEAPPS XML A standard for electronic invoicing developed byTietoEnatorTIEKE Finnish Information Society Development Centre is theclient of this project.UN/EDIFACT United Nations Directories for Electronic Data Interchangefor Administration, Commerce and Transport.Valo A standard for electronic invoicingW3C World Wide Web Consortium. W3C is developing many ofthe technologies used in Web, e.g. XML Schema.XML Extensible Markup Language.XML Schema XML Schemas provide a means for defining the structure,content and semantics of XML documents. In this report,only W3C's XML Schema (XSD) is considered.XSD XSD (XML Schema Definition) is W3C's schemalanguage definition.2 IntroductionThis document is the seminar report for the T-81.161 ICTEC Project Course. The report is about electronic invoicing. The goal of the seminar work is to implement a validation service for different types of XML (Extensible Markup Language) based eInvoices.This work is done in co-operation with the Finnish eInvoice Forum, hosted by TIEKE (Finnish Information Society Development Centre). See chapter 3.2 for more information on the eInvoice Forum.2.1 BackgroundNowadays more and more invoices, especially in B2B (Business-To-Business) commerce, are being transferred in electronic format over the Internet. Currently over 200 million B2B invoices are annually sent in Finland. In addition about 150 million B2C (Business-To-Consumer) invoices are sent. Currently only about four percent of these invoices are transferred in an electronic format. The amount is expected to grow significantly during the next few years, when the electronic invoicing standardization process advances and the solutions become cheaper (eInvoice Consortium, 2005).The total cost of one B2B invoice is calculated to be about 30 EUR, including costs for both sender and receiver. The receiver's part of this cost is about 80 percent. With electronic invoices these costs can be reduced significantly, resulting in industry-wide savings of about 2,7 billion EUR annually (eInvoice Consortium, 2005).2.2 Research ProblemOne of the major problems in electronic invoicing is creating common standards. Traditionally invoice and order information between large corporations has been transferred in EDIFACT form. The implementation of an EDI (Electronic Data Interchange) system is a fairly expensive operation, which reduces the chances of EDI usage for smaller companies.Another problem is the large variety of invoice standards. There are five existing electronic invoice standards in Finland. These are Finvoice (FinnishBanker’s Association, 2005), eInvoice (eInvoice Consortium, 2005), TEAPPS XML (TietoEnator, 2005), PostiXML (Posti, 2005) and Valo (TIEKE, 2005). This report focuses on two questions:1) Is it possible to implement a web-based service that validates XML-based invoices?2) Are there existing validation services for electronic invoices?2.3 ObjectivesThe primary objective of this seminar work is to show, that it is possible to make a web based validation service, which validates against different types of XML schema based electronic invoices. The validator service is meant to provide complementary services for existing web based services owned and administered by the companies involved in this project.The secondary target is to produce a scientific report out of results, which can be used by TIEKE and the companies in the software house working group, and also by other possible interest groups for continuing the work. This is why the report describes some functionality that will not be actually implemented during this project. The service itself will be a demo type of service, which will proof, that it is possible to validate invoices with a web based service.2.4 ScopeDue to limited resources of this course, the validation service cannot be implemented so that it validates against all types of electronic invoices. This seminar work focuses on invoice formats that are specified by an XML schema, such as Finvoice. XML documents validated by a DTD (Document Type Definition) are not handled in this work.The functionality of the service was initially limited to simple document validation, which gives a valid/invalid answer out of the source document. As the software house working group's expectations and needs became apparent during the project, finding tools to extend basic XML schema validation capabilities became part of the scope. These tools could be used to validate the contents of electronic invoices, as well as giving the user some intelligent feedback about the validation process. This seminar work does not focus ontransforming documents from one invoice format to another.The literature review part of this seminar work is rather limited. The purpose of this review is only to give the reader a quick insight to the technologies used and make sure that a similar service has not already been built.2.5 MethodologyThe research method that will be used in our research is constructive research (Kasanen et al., 1993). This is a widely used method especially in software engineering. In constructive research the goal is to create a solution for a practical problem by constructing something novel. This construct can be examined in the light of relevant research publications etc. It should be relevant from both theoretical and practical point of view. In Figure 1, the general research process is shown.We will follow these basic steps also in our research. First, we identify aproblem which, as presented in 2.2, is the validation of XML-based electronic invoices. The main goal of the project is to produce a prototype of a web-based tool for accomplishing this task, which can be used to prove the concept and aid in future development of related products. We will also take a look at what is the state-of-the-art in XML validation, and how these tools might be integrated into our prototype tool for added functionality. In addition to this literature research, meetings will be held with TIEKE's software house working group's members in order to gather additional feedback and ideas from experts in electronic invoicing. These meetings will also act as a focus group validation for our prototype. After the prototype is built, a final conclusion on the findings will be made, and the possible future prospects of the concept will be evaluated.2.6 Structure of the reportThe third chapter focuses on what has been done before by others and how. It opens up the main concepts like XML schema for the reader and answers the second research question “Are there existing validation services for electronic invoices”?The fourth chapter describes the implementation of the validator service itself. It answers the primary research question “Is it possible to implement a web based service that validates XML-based invoices”?The fifth chapter sums up the work that has been done. It tells the reader shortly what was learned about W3C XML Schema based validator services and gives recommendations on how the validation service created in this seminar work could be extended.The references and appendices are described in the end of this report.3 Existing work3.1 Electronic invoice standardsThere are numerous standards for electronic invoicing. Currently there are five existing electronic invoice standards in Finland. These are:Finvoice (Finnish Banker’s Association, 2005)eInvoice (eInvoice Consortium, 2005)TEAPPS XML (TietoEnator, 2005)PostiXML (Posti, 2005)Valo (TIEKE, 2005).Of these standards Finvoice, eInvoice and TEAPPS XML are the most commonly used.Finvoice is the main subject of our study, since its definition is based on XSD, W3C's XML Schema definition. Finvoice has been developed by the Finnish Banker's Association.3.2 TIEKE's eInvoice ForumTIEKE is co-ordinating the eInvoice Forum, which is stated to be a central collaboration and meeting point for the different parties in the field, such as developers, service providers, experts and the users of electronic invoicing. Members of the forums contain significant companies from Finnish industry like BasWare, Elisa OY, Elma Oyj, Enfo, Nordea Bank, OKO Bank, Sampo, Finnish Banker’s Association (chairman 2005), TeliaSonera, TietoEnator, and WM-Data (eInvoice Forum, 2005).3.3 XML Schema technologiesThere are numerous different technologies and tools for specifying XML (see e.g. (Lee and Chu, 2000) for a comparison between different XML schema languages)), with varying capabilities and functionalities. One of the earliest languages for specifying XML documents was DTD (Document Type Definition). A DTD expresses a set of rules for an XML document using EBNF (Extended Backus-Naur Form) grammar1. DTDs have several shortcomings. First of all, DTDs are not XML themselves, which poses problems for XML parsers in handling DTDs. Second, there is no data typing in DTD. Third, DTD doesn't allow the usage of namespaces, which seriously limits the usability of element names.1http://www.garshol.priv.no/download/text/bnf.html3.3.1 XSD (XML Schema Definition)XML schema development started originally at the W3C because of the shortcomings of DTDs used to describe XML documents. Now it is a widely-used language for specifying XML documents. There are several advantages of using XML Schemas instead of DTDs. First of all, XML Schema instances are themselves XML documents. Moreover, XML Schema supports different data types, such as booleans, integeres, characters etc. XML schema also provides a wide range of constraints which can be used check for valid values. These constraints can be e.g. cardinality constraints (minOccurs, maxOccurs), range constraints (minExclusive, maxExclusive etc.) or length constraints (minLength, maxLength etc.).XSD is W3C's schema definition. There are others, of which RELAX NG is one example.3.3.2 RELAX NGThe RELAX NG is a schema language for XML. The RELAX NG specifications have been developed within OASIS by the RELAX NG Technical Committee. RELAX NG is being developed into an International Standard (ISO/IEC 19757-2) by ISO/IEC JTC1/SC34/WG1 (RELAX NG, 2005).3.3.3 SchematronSchematron (Jelliffe, 2001) is a schema language for specifying XML documents, but it differs in basic concept from other schema languages in that it not based on grammars but on finding tree patterns in the parsed document. This approach allows many kinds of structures to be represented which are inconvenient and difficult in grammar-based schema languages. With Schematron, also dependencies between elements can be validated, as can the content of different elements. A good example in the scope of electronic invoicing would be calculating the amounts from invoice rows and checking that the total amount of the invoice matches this sum. This kind of validation cannot be done with XML Schema. Also, Schematron provides an easy way for producing intelligent feedback of the validation process. Schematron isbased on XPath expressions and XSLTs (XSL transformations). Currently Schematron is being developed into an ISO standard. The following picture shows the basic process of Schematron validation:Figure 2: Schematron validation process (, 2005)As the picture shows, the validation process is a two-phase process where first a validating stylesheet is produced based on a basic stylesheet and a schema made by the user. Then the XML document itself is validated using the validating stylesheet. This process is very flexible in that the output can be text, html, or even pdf. This way it is easy to create user-friendly error reports. Users can create their own meta-stylesheets overriding the basic functionality provided by schematron's basic stylesheets.3.4 XML Schema based validators3.4.1 Existing online validation servicesBackground research was conducted in order to find existing online validation services. The research was conducted with the help of popular search engines on the web. The research engines used included Google, Yahoo, and MSN Search. It turned out that there are plenty of online xml validation services2, although with varying quality in features and usability.Invoice specific validation services were not found. Two validation services2See, for example: , or/schemaValidate.htmlexceeded the masses in overall usefulness: and . Both of the services are able to list all the errors at one go. This is a feature most of the lower quality services lacked.W3C is providing high quality online validation services for most of their own standard document types. The service is widely known and used. As mentioned, W3C concentrates in supporting the adoptance of their own standards and hence the services are not ideal for general purpose xml validation.Validome provides an excellent online validation service for any type of xml document. The service has been build to assist web publishers to get their sites standards compliant, but it also suits very well for custom xml validation. Besides the web interface Validome also offers a machine interface for automated xml validation.W3C is providing the source code for their validator services. The validators for different W3C standards are implemented with varying development tools such as C, Perl, and Python. Unluckily none of the packages were written for the J2EE environment, which the project group is using. Validome is not publishing any source code of their service.3.4.2 Existing useful software packagesIt is known that XML schema validation features do exist in development tools and also as online services. Research was made to find existing packages which could take the validation to the next level.The use of Schematron rule engine for validating the contents of the invoices would seem ideal. Instead of using static coded rules for the known invoice formats one could feature a more configurable and maintainable rule file. Combining the usability of Validome's web interface and Schematron's expression power would certainly create an excellent basis for custom online invoice validation service.3.5 Invoice validatorsIn the background research powered by top search engines on the web, nolinks to existing invoice validators were found. As mentioned, Validome is a multi-purpose validation service and thus also capable on validating invoices defined with W3C XML Schema. No validation services featuring tree pattern matching capabilities were found.Besides online search engines, several3 scientific databases were queried to find XML validation related research papers. The closest matches were studies where XML validation was committed in a special situation like incremental validation (Balmin et al., 2004) or validation of streaming XML (Segoufin et al., 2002). No XML invoice specific research papers were found.4 Validator serviceA simple validation service was built to answer the research question 1. It shows that it is possible to build a web-based service, which validates XML-based invoice documents properly. The functionality of this service was broadened to better meet the wishes and requirements from the software house working group. This chapter describes the system built.4.1 RequirementsThe basic requirements for the validation service were specified by TIEKE, and they are as follows:1. The service can validate XML invoices against a given schema definition.2. The service can provide meaningful feedback of erroneous elements.3. The service has to be easily usable over the Internet.During the project, it became evident that simple XML validation is rather straight-forward, and doesn't provide anything new in comparison with existing technologies, so this was extended to include Schematron validation in order to offer extended validation capabilities, which were desired by the software house working group. Java XML parsing techniques do not offer a simple way for giving useful feedback in the case of validation errors. With Schematron these possible error statements can be maintained in the Schematron schema3 ABI Inform, ACM Digital Library, Computer and Information Systems Abstracts, andComputer Science Bibliographiesfile, and they are far easier to manage.The requirements of the simple validator can be summerized into a single use case:UC1 Validating an invoiceSummary Validator checks whether his / her XML invoice document is valid or not.Stakeholders Validator – the person who validates the invoice document Preconditions Validator has logged on to the validation forum and has the validation page (Figure 1) on his / her screen. An XML Schemaand the corresponding Schematron schema exist on theserver.Event order 1. Validator copy-pastes the content of the invoice documenton to the ‘Verkkolasku’ –text box.2. Validator chooses the schema against which he / she wantsto validate the invoice3. Validator pushes the ‘Validoi’ button4. Validator gets feedback onto his / her screen whether theinvoice document was valid or not. If the document wasinvalid5. If the document was invalid, the user can view the errorreports resulting from the xml schema validation andschematron schema validationExceptions -Final state Validator knows that his / her invoice document is valid or that it was invalid for a specific reason.Table 1: Use Case 1 - Validating an invoice4.2 FunctionalityTo meet the requirements common web application architecture was considered. Figure 2.1 illustrates the setup where a user, in any networked location, is able to connect static and dynamic services through the public Internet. For the required functionality, online XML invoice validation, this setup would be sufficient. For an advanced service there could also be a relational database included which the application server could use as persistent data storage.Figure 2.1: Common web application architectureThe validator service was implemented on a web based user interface using JSP (Java Server Pages) technology. A web based user interface was selected for demonstration purposes. For the validator service itself it makes no difference whether it is implemented on a web based user interface or some other e.g. command line interface.The user interface, which was implemented in Finnish for the client, is fairly simple. On the first page, which is shown in Figure 1, the user gives the input arguments for the validator. The first argument is the XML invoice document to be validated in text form. The second argument is the schema that the invoice is validated against. After giving the arguments the user presses the ‘Validoi’ button, which starts the validation process. There are actually two different validation processes. First one is the validation of the invoice against its schema specification. Second validation validates the invoice against its Schematron schema. Both reports can be reviewed after the validation process.Figure 3: Giving parameters of validationThe second page displays the results of the validation. It gives the user information whether the invoice given was valid or not. The result page is shown in Figure 2. This page provides links to validation reports generated from the XML Schema and Schematron schema validation processes. In the Figure 2 a situation can be seen where simple XML validation has been passed, but Schematron validation has failed. This can be a result from a situation where Schematron validates e.g. that an element containing the sum of certain elements equals to the actual sum of the elements. This kind ofvalidation cannot be performed using XML Schema.Figure 4: Result of validation4.3 ArchitectureThe basic architecture of the system is shown below in Figure 5. It is a simple architecture, consisting of two different layers. The presentation layer consists of JSP pages, which in turn access data using the classes in thefi.tieke.validator package. DatabaseManager class is responsible for handling database queries. XmlValidator class is used to validate an XML document against a given schema. SchematronValidator handles validation using the Schematron tool. ErrorHandler class takes care of handling errors during validation. This is only a very crude architecture. All these components werenot implemented during the project.Figure 5: Basic architecture of the system4.4 Technical details4.4.1 Server requirementsThe development environment of the validator differed much from the actual production environment, to which the validator was installed on, but this didn’t cause any problems. The validator service runs in both Linux and Windows environments. A computer with 466 MHz processor and 128 MB main memory was enough to run the service. The largest document that the validator was tested sized 109 KB, with which the validation took only a few seconds. Example invoices can be found e.g. from the Finnish Banker’s Association home pages (Finnish Banker's Association, 2005).As the validator service is used through the web interface, a web server has to be installed on the server machine. A more detailed description of both development and production environment is given below.Operating system Microsoft Windows 98 MECPU Intel Celeron, 466 MHzMemory 128 MBHttp Server JBoss v. 3.0.6Installed Java packages Java 2 Platform, Enterprise Edition v. 1.4;Java 2 Platform, Standard Edition v. 5.0Table 2: Development environmentOperating system Linux 2.4.21-4.EL (RHEL AS 3v4)CPU Intel Pentium III 1133MHzMemory 1024MtHttp Server Apache Tomcat v. 5.5.7Installed Java packages Java 2 Platform, Enterprise Edition v. 1.4;Java 2 Platform, Standard Edition v. 5.0Table 3: Production environment4.4.2 Client requirementsSystem requirements are explained in section 4.1. To use the service, a client only needs an Internet connection and a web browser to use the service through a web based user interface.4.5 Testing the simple validatorFor the simple validator it makes no difference whether the XML document that is validated, is an invoice document or some other type of XML document. Therefore a simple XML instance and XML schema instance were created for testing purposes. Testing was done by running the simple validator with different kinds of inputs, both valid and invalid. The different inputs formed five test cases, which were defined by the project group. The XML instance was modified for the purposes of each test case. The modifications are given in each test case separately.The simple schema instance used in testing:<?xml version="1.0" encoding="UTF-8"?><xs:schema xmlns:xs="/2001/XMLSchema" elementFormDefault="qualified" attributeFormDefault="unqualified">name="rootElement"><xs:element<xs:complexType><xs:sequence>type="xs:string"/>name="name"<xs:elementname="value"type="xs:int"/><xs:element</xs:sequence></xs:complexType></xs:element></xs:schema>The simple XML instance used in testing:<?xml version="1.0" encoding="UTF-8"?><rootElement xmlns:xsi="/2001/XMLSchema-instance"xsi:noNamespaceSchemaLocation="schema.xsd"><name>Invoice</name><value>57</value></rootElement>4.5.1 Test casesTC1 Validating against a valid documentSummary: In this test case the validation was done against the originalXML instance, which was valid.Input: The original XML instanceValidExpectedoutput:Actual output: The document was valid.Table 4: Test case 1 - Validating against a valid documentTC2 Validating against a not-well-formed documentSummary: In this test case the original XML instance was modified intoa not-well-formed format.Input 1: The element <name> was changed into <nam>Input 2: The element <name> was changed into <nameNot well formedExpectedoutput 1:ExpectedNot well formedoutput 2:Actual output 1: The document was invalid.The end-tag for element type "nam" must end with a '>'delimiter.Actual output 2: The document was invalid.Element type "nameInvoice" must be followed by eitherattribute specifications,">" or "/>".Table 5: Test case 2 - Validating against a not-well-formed documentTC3 Validating against an element with false simple type。