Telecommunication data mining_pic

Telecommunication Data MiningMarijo NižetićCroatian Telecom - HT TKC SplitVinkovačka 19, 21000 SPLITPhone: 021 351-370, mob: 098 764-898, E-mail: marijo.nizetic@st.hinet.hrAbstract - The essential results of the telecommunication management may be presented by the sets of collected, timed, and in the relation databases stored data, readily fitted to data mining and aimed for the efficient telecommunication network exploitation. The collected data, divided in the five data sets, according to the recommended global process of data assembling, were researched on both lower and higher layers of OSI model. First, on a lower layers the data were purely analyzed by means of counting process, targeted to stochastic processes in both, model identification and parameter estimation, in finding out Bernoulli, Poisson, Markov, and similar processes over the all mixed data sets. Second, in data mining on a higher layers, a methodology of Control Theory was applied, to forecast, forced to bring out, and prevent the inconsistencies among the managed data.I. I NTRODUCTIONIn a time, the system transfers matter, energy, or information. The aim of communication system is an exact transfer of information in a real time. It is possible to look at the four aspects of information content: syntactic, aesthetic, semantic, and pragmatic. Two latest ones are target of presented research. It is possible to say that under certain conditions, an aesthetic aspect is covers all of the remaining aspects. Under the information transfer there is necessarily to control communication system in an efficient manner, to satisfy completely all of the aspects mentioned before.Among nowadays communication systems the discrete (stochastic) event systems play a prevalent role. A Discrete Event System, (DES) is a discrete-state, event-driven system; that is, its state evolution depends entirely on the occurrence of asynchronous discrete events over time [1]. If attribute stochastic is taken in consideration, the above definition broadens in the sense stochastic discrete events. International Standard Organization (ISO) had provided for the decomposition of managing aspects with its Open System Interconnection (OSI) model divided in the seven management layers. More rough division was done on lower (network functions) and higher (user' functions) layers.The DES both concept and model are common for both lower and higher OSI [3] model layers. According to that concept, it is possible to model (enclose behavior) the communication system from data cell, buffer, and queuing system on one side; over fuzzy logic, inference machine, and a fuzzy implicate relation in the middle; to knowledge database, intelligent interface, and artificial intelligence on the other side.The nature of classification is derived from nature of essential communication event, i.e. call (communication) and its essential parameters: establishing, duration, and disassembling the communication links. Most system definitions are using just the two highlighted properties: interaction among components, and association of "functions" that are intended to perform. The realization of high quality control is unthinkable without feedback property [6].The OSI model is an implementation of the four information aspects mentioned before, in a real communication environment. It is important to note a possibility of using the same methodology for describing the control activities on both lower and higher OSI model layers using the Control Theory principles. It is possible to apply the same methodology using direct mathematics description on the both, communication processes invoked inside a communication network, and control processes realized inside the management system.Now, the Control Theory is not ready to capture semantic and aesthetic aspect of information content, but concept of the state-space is quite fitted to the structure of modern (tele)communication systems. As earlier mentioned the essential event is a call, described with a huge set of parameters through the database in a real time. The parameters are assembled to the abstract entity data sets describing a call. The predominant sets of data belonging to alarm, charging, traffic, administration, and logistic support are of the computer data kind. Every relevant event is adequately timed, and has associated the corresponding state-space.According to the definition, a communication system is a stochastic system, if at least one of its output variables is a random variable. The state of a stochastic dynamic system defines a random process, which behavior can be described probabilistic only. Contrary deterministic system, which state x(t) can be evaluated by the given input u(t) for all t> t0, in a stochastic system the state in the time, t, is a stochastic vector ready to derive a probability distribution function.The model concept of the communication system so posed, facilitates the system analyzes of dynamic behavior in both transition- and steady-state, in presence of irregular and non deterministic stochastic processes, that could not be describe with mathematical equations. For example, in monitoring the communication service set, there is an interesting question: Can it be possible to reach either the particular or the set of states from the determined initial conditions in any time? It is very important control question that must be solved at beginning of the system building, through the telecommunication services project process, to avoid the cases of blocking, deadlock, and livelock. Problems just pointed out are only marked and will not be considered in this paper.The target of this paper is to describe researching of the stochastic properties communication sub-processes, aiming to efficient communication system management. At the highest level of abstraction the modern communication system can be presented by the model essentially posed with a timed relation database structure. Such a structure presents the vector as realization of a stochastic process, mathematically. In practice, the parameter data have been extracted from the process data, and have been assigned to both local and Intranet network of agents at information level. Reasonable question is whether only one realization of the stochastic communication process is sufficient. The answer is positive and proved later in the paper, based on the analyzed data collected by PUNOa1program through the years.II. (TELE)COMMUNICATION MANAGEMENT Telecommunication management comprises of using both, algorithms and feedback. More than ever the control technology includes modeling, sensing, actuation, and a real time calculation in a common task. A great active attention is devoted to control both dynamics and stability of the controlled communication system with a large support of computer systems and computer networks. In that way, the own controlled communication network becomes integral part of the computer network. As a consequence it demands an integrated management on both lower and higher layers of OSI model. However, the active attention of control action must increase to achieve a precision.Control has directed the systems to behave in a desired manner. For that reason, the feedback loop is used to change dynamics and handling with an uncertainty of controlled system, as far uncontrollable uncertainty leads to the catastrophic failure. The statistic analyses integrated into a program support of the physics (the control) systems, are used also, to achieve direct both, reconfiguration and adaptation to the real environment.In all of these areas a common feature is that the requirements on the system level far exceed the achievable reliability of its components. This is precisely the fact, where feedback (in its most general sense) plays a central role, since it ensures the system to achieve a goal, through the corrective actions, based on sensing its current state. The challenge in the field is moving from the traditional point of view the control systems as an interconnected set of components, to realizing control systems as a heterogeneous collection of both physical and information systems, with intricate both interconnections and interactions.Input/output approach and control model approach in the state-space allows: disturbance rejection, model reduction, robust interconnection between subsystems, and software interacting with the physical environment, done in a real time. A central role is given to the way in which computer considers the controlled communication system entities either near or far peripheral units, providing the flexible system control.1PUNOa is an achronym in Croatian language of "Program upravljanja, nadzora, održavanja i administriranja"2PUNOa is an achronym in Croatian language of "Program upravljanja, nadzora, održavanja i administriranja"A. Sampling and digitizationTo come nearer the Control Theory in controlling the communication network processes, there is a need to observe the two terms closer: digitization and sampling. From the mathematical point of view these terms have to possess the same meaning as corresponding terms that are used in an analog signal digitization, supported by the sampling theorem. However, there is a necessary to prove that equality is valid in the field of managing the communication processes at the higher OSI model layers. Let's pose the operation thesis: having "digitized" information belonging to the higher OSI model layers, we can reconstruct process path in the state-space. Then, we can observe the law of bending the corresponding curve, and allow using the predictive method of the Control Theory in both forecasting the behavior and prevent acting to the controlled processes.Among all counted and classified processes, the part of the charging process will be described later as a paradigm (as it is managed by PUNOa program).B. Hierarchy of the communication processesEssential parameters of the communication process are: •beginning, duration, ending;The extended parameters set (the subset) of the communication process are:•terminal kind, switching type, kind of transmission system, service class, …;•speed, quantity, delay, …;•physical information shape, frequency, signal/noise, modulation, multiplexing, signaling, programsupport, …;•service policy, the costs, …;•charging, alarm, traffic, administration, …;•modeling, managing, ….The essential process parameters (subset) for managing the communication processes are:•control, monitor, maintenance, administrating, logistic support, ….According to essential process of communication, every parameter layer is converted into the new sets of the sub-processes with corresponding sub-layers taken as the parameters, which again, by means of decomposition process, shapes the sub-processes with associated the new parameters, and so on.C. Organization the sets of controlled dataBy the interruptedness operation of the PUNOa program, the controllable data sets are collected and stored into a few essential data sets. The aim is realization the process of collecting, managing, storing, and taking an action according to the sharp conclusion of the PUNOa program module. The PUNOa program is spending the available unoccupied time, to make essential stochastic analysis, and associate such created data to databases. Some other more complex analyses are performed by external auxiliary the programs and their results have been added to the existing data in databases. These data belong to the sets of: charging, alarm, traffic, administrative, and logistic support data.1)Charging data: The essential form of the charging data organization is a daily database form that includestimed descriptions of the cumulative counter state for every subscriber. An auxiliary database that describes the both state and behavior of every subscriber in time up to instant t0, is attached to such daily charging database. Both data associated to particular switching node and statistics derived from them are at our disposal. Especially for efficient exploitation, information about extreme value of a particular charging counter is very important, directed to detect the possibility of an error either in switching or information domain. Regarding the importance of economic component in the large communication environment, a special attention is given to the charging domain.All the collected data are checking manifold, so the possibility of error is very minimal, and empirically is found about 1.7⋅10−7with a negative trend. The essential checks are compared with: already existing data, computed possible values, parallel data from other sources (toll ticketing), and analyses of all the other data sets (alarm, charging, traffic, administration), that describe the same event - a particular communication link.2) Alarm data: The thesis of PUNOa program is "Alarm is everything" [4], so the alarm categories include all kind of irregularities found in all other data sets, including alarm data. Such an approach brings out a need of the category dynamic change for the existing vendor's alarm level settings, and introduces the own categories of the alarm stages with the same possibility of dynamic change. Also, these activities are incorporated in the code lines of PUNOa program. The missing activity is feedback loop back to the communication equipment vendors in the sense of change the factory defined alarm categories according to the needs of adaptation in exploitation, but it is left out for some other paper.3) Traffic data: On the higher layers of OSI model only, the importance of traffic data is pushed out to the background by: developing and introducing computer supported communication systems, miniaturizing of electronic elements and components, increasing the information channels throughput, and decreasing the consumed energy. On the lower layers of OSI model, particularly at introducing broadband communication network, the importance of traffic data has been increased, and until now play the prevalent role at the science research activities.In the PUNOa program [5], manipulating with the traffic data is being obtained according to the general principles of the traffic communication data tooling, but one more component is added. Namely, according to the general principles pointed out above, the data, a priory do not belong to the traffic data set, are analyzed and processed, but the obtained results have the traffic characterization. Another reason is that traffic data processing on the data cell layer, associated with the true traffic data on the higher layers of the processing, gives a possibility to realize of a powerful tool, to meet the control activities of networked control systems (NCSs) [8]. The PUNOa program is the right example of such NCS system, however without insight in the traffic data on the data cell level. PUNOa program has to reach such a status, however the time will confirm a soundness of demand like that. A needed environment (Intranet) already exists.4) Administration data: The administrative domain data presents the input data for more communication processes. In the PUNOa program, these data are under alarm status control, and in a praxis it proves to be very important in resolving extreme and inconsistent states among charging data. For example, due to the feedback loop in a daily data processing, the fuzzy logic module of PUNOa program has both discovered and proposed the way of resolving the problem of the wrong administrative data entrances inside the regional switching node tables. In the future the process of managing the information, an administrative component will participate directly in both, changing the system data of switching node and feedback realization, to confirm the soundness of involved data. At that moment, status of all the technical support departments have been served the communication system management, will be marked by question mark.5) Logistic support data: It is important to emphasize the significant equal treatment of the logistic support data as a fundamental data, due to existing relationships. It was not a case until nowadays praxis, but today the praxis is changing. Regarding the PUNOa program, these data sets have been processed off-line as demonstrated later in the paper; the program has occasionally applied the achieved results. The data belong to the logistic support data set, were collected and stored in the relation databases by means of PUNOa program, and in a series of published papers were described in details.III. DATA MINING AND ACHIEVED RESULTS A. Data miningProcessing the collected above described data through the years, some directions of the data analyses were fixed by discovering the ways of formalizing the relationship among them. In short will be described some details of approach to managing the collected charging data, resul-ting in some new terms and stochastic processes definition. The analyses were performed has proved the validness of joining the meaning of one representative realization of a new coined process to the particular database record. TheFig. 1: The block scheme of data mining process.As already pointed out the general OSI model is divided to higher and lower layers. It fact is confirmed according to the consequences on the processed data collected by the PUNOa program.B. Higher OSI model layersOn the higher OSI model layers PUNOa program uses the data stored in the relational databases for the purpose to bring out the conclusions and propose the right actions on the controlled system. For that purpose the program isorganized as the fuzzy conclusion and decision making machine. The thesis that communication system can be modeled by both mapping and integrate managing the data belongs to the different abstract communication entities on the same information level, is proved by the theoretical part of the PUNOa program.Its name is SPIS3(Set of Pursuits in an Information System), and it gives the answer to the questions, posed in [3, chapter 19], asking for answers to the new questions (8. layer of OSI model, for example), and opens the new research fields (intelligent interface). The higher layers of OSI model are adapted to the research of semantic and pragmatic aspect of the information content.1)Fuzzy logic: Except processing the data stored in a database as the essential elements of a discrete mathematics process model before described, PUNOa program uses already highlighted fuzzy logic module. Joining the corresponding knowledge and action databases, program is analyzing the data on a higher OSI model layers, brings out the conclusion, and takes the action in a meaning manner. In that sense PUNOa program presents the set of the networked intelligent agents, showing how to build in an artificial intelligence in a communication environment.2)Artificial intelligence: The fact, the databases as strictly arranged as the sets of data, is essential to build in the elements of artificial intelligence into communication management system, as forecasted in [7], is proved by the PUNOa program.3)Intelligent interface (input data filter): Very important part of the communication management is an intelligent interface that includes both fuzzy logic and artificial intelligence in a smart manner. The first interface role that has been done after management process, is linking the data sources to the relation databases. However, the database placed behind the interface has been possessed a fixed structure in a high grade, but telecommunication environment posed in front of interface, is changeable in an unpredictable way and is infected by the noise. The second interface role is an adaptation according to applying the different vendor's software equipment. These are very sophisticated demands; not realizable without the interface intelligent component supported by the knowledge database.C. Lower OSI model layersOn the lower OSI model layers PUNOa program is capable to identify and to verify of belonging the particular process to some of discrete mathematical models (e.g. Bernoulli, Poisson). It includes the parameter estimation of the observed process, because the data stored in databases are readily available for managing. The off-line parameters estimation is necessary to build-up better quality the program modules, aiming to achieve final result i.e. on-line parameters fitting in a real time for all the processes. Thus, although the collected data belong to data sets of the higher OSI model layers, simply, they can be aimed to use for the purpose of syntactic aspect of the information content.3 SPIS … acronym of: Skup poslova informacijskog sustava (in Croatian)4 SPIS … akronim: Set of Pursuits in an Information System (Engleski)1)Processes: It is a general proposition [1, chapter 6.] that many communication processes usually belong to the set of Poisson processes. The results of the PUNOa program verify that proposition up to very high segmentation for all the possible processes raised from the collected data, by shaping the new unknown communication processes (described in a progress).2)Statistics: Working on-line, PUNOa program statistically analyzes both collected and treated data, and the results are been stored in the relational databases. Although, some statistical data were not used for a long time period, both storing and saving them had shown to be the right choice, nowadays finding usefulness in the process analyzes. One of the examples is information about parameter named the number of active subscriber in a particular switching node through the measuring time period. Now it is shown the parameter can be dynamically shaped to provide a mathematical process description, useful to forecasting behavior of the observed process with a great precision grade.D. Control Theory and the new research fieldsAll the facts mentioned in this chapter, prove several thesis published in [6] aiming to extend a field of action belong to the classic Control Theory, to the new areas using the new principles of modern Control Theory in the state-space. (It is of interest to note that in quoted material, the field of communication is not pointed out as a possible research field.) Current approaches in software engineering and control synthesis are not sufficient enough; it is essential to merge ideas from both computer science and Control Theory, as an example provided by the PUNOa program. The strength of the field relies on our close contact with rigorous mathematics, increasingly important in the future. Very important methods to consider complex composed processes in a communication field are feedback and state-space including matrix tool.1)Feedback: Control is considering the way in which the managed communication system should behave in a desirable fashion. As does feedback, as a tool for tracing and changing the dynamics is a very powerful tool for managing the uncertainty [2]. Feedback can be used to stabilize a system, speed up a transient response, improve the steady-state characteristics, provide a disturbance rejection, and decrease sensitivity to the parameter variations. Proportional feedback reduces the errors, but high gain may destabilize the system. Integral control improves the steady-state error and provides robustness with respect to parameter variations, but it also reduces the stability.2) State-space: Since it is particularly well suited to use the computer techniques in a communication system control, state-space design is increasingly studied, and today is used by the control field researchers. The use of the state-space approach has often been referred to as modern control design. Stochastic process X(ω, t) is using to describe state of the system. If variable ωis fixed, the resulting deterministic timed function is known either as path or as realization of the stochastic process. A path can be imagined as a consequence of particular random experiment that describes the state of system in time. On the other hand, if variable t is fixed, we have got random variable identified as a random state description of thesystem in a time, t (for example the daily state of particular charging counter in PUNOa program).3) Complex structured processes: It is very clear that input, output, and state of the communication system data structures, suggest the usage a Control Theory method applied to all kinds of the classical SISO and MIMO processes. However, a database structure already possessed a matrix lattice, one of the essential tools to follow up the modern Control Theory.E. Ergodicity of the charging data sourcesIf a statistical property of the data source does not change in time, we say either the source has a stationary property or the source is invariant, otherwise is non-stationary. Most real communication data sources are non-stationary. The processes in the communication system belong to such class of the processes.Particularly, in the communication research field, the stationary sources generating messages (data) according to, so called, the Bernoulli's law of large numbers, take attention. Such the sources have no more different working regimes, and are named as ergodic. Most real stationary processes can be considered as ergodic processes, at least with some acceptable restrictions. The important ergodicity consequence is that the source statistical properties can be analyzed based on of observing the only one process realization, i.e. based on the only one timed signal generated by the source.The communication source that has only one working regime is an ergodic source, and the data array generated by it keep the same features, i.e. for a sequences long enough, statistical parameters lead to the same limit value regardless what is the process realization about.This feature, far away from reality at the first sight for most statistical methods of the source consideration, is based on one and only one data sequence. We say that statistical parameters evaluation of the sources (processes) can be considered upon only one of executions with a probability equaling 1. The ergodicity property of some source can be considered as a property of statistical homogeneity of the flow discrete data long enough.Is the charging stochastic process stationary, i.e. does it possesses ergodicity that can be evaluated based on the only one and finite sequence of the counter state? To get a positive answer to that question, it was needed to research the auto-correlation function of the charging process in the different time intervals, for the same subscriber number. Also, it was needed to research the correlation function in the same charging periods for the different two counters. Instead of the counter state, it could be possible to analyze the times between the charging events, but it is left out for the further research activities. All other analyses on the counted processes were done successfully.The ergodicity property, based on one process realization in time instant, enable to conclude on stochastic properties of the communication processes, has a special meaning, because in practice one timed charging function per one subscriber is available. Based on the fact that ergodic function is a consequence of discipline subscriber's behavior in monthly periods, the arithmetic mean per sequence was evaluated, and so, the ergodicity of charging process was proved.1) Typical communication stochastic processes: A stochastic or random process is simply the collection of random variables indexed through some parameter (which is normally thought of as "time"). Let Ω= {no charging, charging} be a space for describing the random state of charging process in communication system. By mapping no charging into 0 and charging into 1 we may define a random variable X(ω),ω∈Ω, which takes the values 0 or 1. Thus, X(ω) = 1 means the system is charging, and X(ω) = 0 means the system is no charging. Next, suppose we wish to describe the state of the charging process once a day every day, and let k = 1,2, … index the days. Thus, X(ω, k)is a random variable describing the state of the defined charging process on the k th day. The collection of random variables {X(ω, 1),X(ω, 2),…,X(ω, k),…} defines a stochastic process.The Poisson simple counting process is modeling much communication processes, accurately, including process described above. The Poisson process is an essential building block in DES both stochastic modeling and analysis, because it permits us simplify the Markov models to obtain Markov chains more suitable for analyzes. The Poisson process counts the events, occurs in a very random but time-invariant way, so the assumption (A1) through (A3) in Chapter 6 of [1] are satisfied indeed.2) The charging data sets ready to analyze, were collected by PUNOa program, and stored in the databases under general name structure TARIFAmt.ext. It involves about 30 timed realizations for every subscriber belonging to the particular switching node. Research activities on the collected data bring out a definition for the new processes: the daily charging and at least one connection made. The findings relative for the leader process belong to the Poisson process class, and for the follower process belong to the Bernoulli process class. Both the processes satisfy assumptions (A1) through 4(A3) mentioned above, and obtained typical graph shapes are shown on Fig. 2, and Fig. 3, respectively.Fig. 2: The Poisson distribution for m closer 1Fig. 3: An example of charging data obtained from empiric data。