第4讲-DW-Operations

d in multidimensional array to remove the need for storing the dimension values.
– Step 2 The array is transformed into a linearized array by an array linearization function – Step 3 The linearized array is compressed by mappingcomplete compression method.
2013-8-27
HD-ITR
10
4.1.1 G-Aggregation Algorithm
• • • • Input: Compressed dataset R Output: Agg(R, S, F) “G" algorithm can be used in all situations Consists of two phases.
S(B, C;sum( M))
aggregation
R(A, B, C, D; M)
R(B, C, A, D; M)
2013-8-27
HD-ITR
12
Implementation: Compute Agg(R, {Loca}, Sum)
Loca. PN L1 p1 L1 p2 L2 p1 L2 p2 L3 p2 L4 p3 L5 p1 Amount 500 200 300 100 600 400 200 PN p1 Trans. p1 p1 p2 p3 p1 p2 L1 500 200 p2 L2 300 100 p2 L3 600 p3
2013-8-27
HD-ITR
18
4.1.3 Prefix-Aggregation
• Prefix-aggregation is based on the following feature:
– There are common prefix in dimensions serials of the aggregation result and the original data set.
2013-8-27
HD-ITR
15
4.1.2 M-Aggregation Algorithm
• Input: Compressed dataset R • Output: Agg(R, S, F) • Steps
– All temporary results are kept in memory; – When a new record is read, memory is searched to determine if there exists the corresponding temporary result with the new record; – If exists, the value of measure attribute of the new record is aggregated with the corresponding temporary result; – If not, the new record is write to memory as a new temporary result.
2
Dimension order is (Location,PN)
500, 200, 0, 300, 100, 0, 0, 600, 0, 0, 0, 400, 200, 0, 0
3
Header Comp. File 500, 200, 300, 100, 600, 400, 200
HD-ITR
2, 1,
4,
3,
5,
6,
7,
8
2013-8-27
9
4.1 MOLAP-Based Aggregation
• • • • G-Aggregation Algorithm M-Aggregation Algorithm Prefix-Aggregation Algorithm Infix-Aggregation Algorithm
2013-8-27
M M 2 2 3 3 3 3 4 4 5 5 4 4 3 3 B C Sum 1 1 2 1 2 2 M 9 10 5
1 1 2 2 2 1 2
1 1 1 1 2 1 1
1 2 1 2 1 2 1
HD-ITR
7
4.1 MOLAP-Based Aggregation
• Compress the dataset
• Definition of aggregation Agg(R, S, F)
– collapse away dimensions S to obtain a more simple dataset – For example, Agg(R, {A, D}, Sum)
A B C D 1 1 1 1 1 2 2
4
Chapter 4 Operating Algorithms
• • • • 4.1 MOLAP-Based Aggregation 4.2 MOLAP-Based Cube 4.3 ROLAP-Based Cube 4.4 Iceberg Cube
2013-8-27
HD-ITR
5
4.1 MOLAP-Based Aggregation
Amount 500 200 300 100 600 400 200
PN Loca. Trans. p1 L1 p1 L2 p1 p2 p3 p1 L5 p2 L1 L1 500 200 p2 L2 L2 300 100 p2 L3 L3 600 L4 400 p3 L4
L5 200
Amount 500 300 200 200 100 600 400
L4 L5 400 200
Loca. Amount L1 500 L2 300 Agg L5 200 L1 200 L2 100 L3 600 L4 400
PN Amount p1 1000 p2 900 p3 400
2 500 1 200 4 300 3 100 5 600 6 400 7 200 8 Amount B. M
2013-8-27
HD-ITR
8
Loca. L1 L1 L2 L2 L3 L4 L5
PN p1 p2 p1 p2 p2 p3 p1
Amount 500 200 300 100 600 400 200
money 5000 800 3000 500 6000 8000 4000
p1 p2 p3
1
L1 500 200 L2 300 100 L3 600 L4 400 L5 200
• • • • G-Aggregation Algorithm M-Aggregation Algorithm Prefix-Aggregation Algorithm Infix-Aggregation Algorithm
2013-8-27
HD-ITR
6
4.1 MOLAP-Based Aggregation
2013-8-27
HD-ITR
17
4.1 MOLAP-Based Aggregation
• • • • G-Aggregation Algorithm M-Aggregation Algorithm Prefix-Aggregation Algorithm Infix-Aggregation Algorithm
1 500 1 2 6 300 2 2 Sort 200 5 CP 2 6 7 200 6 7 8 100 7 7 14 600 8 8 5 400 14
HD-ITR
Comp. HD File
500 300 200 200 100 600 400
Amount
13
Loca. PN L1 p1 L1 p2 L2 p1 L2 p2 L3 p2 L4 p3 L5 p1
School of Software Engineering
Data Warehouse and Data Mining
Yang Yan
Part 1 Data Warehouse
2013-8-27
HD-ITR
2
Contents
• Data warehouse
– – – – – – Introduction Data model Data Storage and indexing Operating Algorithms Query Processing and Optimization From data warehousing to data mining
2013-8-27
HDComp. File
Transposition
1 2 4 5 8 12 13
(L1, p1) (L1, p2) (L2, p1) (L2, p2) (L3, p2) (L4, p3) (L5, p1) RL (L, P) L (P,L)
500 200 300 100 600 400 200
– Phase one: transposition phase
• Transposes the dimension order of R into a favorable order so that the aggregation can be easily computed.
– Phase two: aggregation phase
2013-8-27
HD-ITR
3
Data storage and indexing
• Summary
– Data storage
• ROLAP • MOLAP • HOLAP