I／O Strategies for the T3E
合集下载
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
FFIO considerations
• Examples above use an unblocked file structure, normal Fortran files are blocked. To read the file without the global or bufa layers you must use assign -s unblocked f:filename
additional flexibility
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
Cray FFIO library
• FFIO is a set of I/O layers tuned for different I/O characteristics
• $TMPDIR
– fast – purged at end of job or session – shares quota with /usr/tmp
• $HOME
– slower – permanent, backed up – check quota with quota (usually 2Gb and 3500 inodes)
• bufa is an asynchronous buffering layer • performs read-ahead, write-behind • specify buffer size with -F bufa:bs:nbufs where bs
is the buffer size in units of 4Kbyte blocks, and nbufs is the number of buffers • buffer space increases your applications memory requirements
changed by calling GLIO_GROUP_MPI(comm) • specify buffer size with -F global:bs:nbufs whБайду номын сангаасre
bs is the buffer size in units of 4Kbyte blocks, and nbufs is the number of buffers per PE
• bufa and global do not allow backspace, or skipping over a partially read record. You can allow this behavior by using the cos layer in addition to bufa or global, but then setpos doesn’t work. assign -s cos:128,bufa:128:2 f:filename
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
More on FFIO
• There are many other FFIO layers, some pretty obscure
– cache and cachea layers, good for random access files
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
Types of I/O
• Language I/O: Fortran or C (ANSI or POSIX) • Cray FFIO library (can be used from Fortran or C) • MPI I/O • Cray extensions to Fortran and C I/O (mostly for
– Language I/O – MPI I/O – Increase language I/O performance with FFIO library
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
I/O Strategies - Shared files
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
global FFIO layer
• global is a caching and buffering layer which enables multiple PEs to read and write to the same file
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
File positioning with the global FFIO layer
• Positioning of a read or write is your responsibility • File pointers are private • Fortran
ffwrite
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
The assign command
• the assign command controls
– controls which FFIO layer is active – striping across multiple partitions – lots more
• use the FFIO library bufa layer to improve performance for all unformatted sequential Fortran files assign -F bufa:128:2 g:su
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
• scope of assign
– File name – Fortran unit number – File type (e.g. all sequential unformatted files)
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
T3E Overview
• T3E is a set of Processing Elements (PE) connected by a fast 3D torus.
• PEs do not have local disk • All PEs access all filesystems equivalently • Path for I/O generally looks like:
compatibility with PVP systems)
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
I/O Strategies - Exclusive access files
• Each PE reads and writes to a separate file
• Buffering of data (configurable size) • Caching of data (configurable size) • Available to regular Fortran I/O without reprogramming • Available for C through POSIX-like calls, e.g. ffopen,
I/O Strategies - Communication and I/O PE
• One PE coordinates reading and writing and communicates data back and forth between other PEs via message passing
• if one PE has already read the data, an additional read request from another PE will result in a remote memory copy
• file open is a synchronizing event • By default, all PEs must open a global file, this can be
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
I/O Strategies for the T3E
Jonathan Carter NERSC User Services
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
– Use a direct access file, and read/write(rec=num) – Use Cray extensions setpos and getpos to position file pointer
(not portable)
•C
– Use ffseek
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
– Language I/O – MPI I/O – Increase language I/O performance with FFIO library (C must use
POSIX style calls)
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
assign Examples
• To see all active assigns assign -V
• To remove all active assigns assign -R
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
bufa FFIO layer
– user buffer space – system buffer space – I/O device buffer space
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
Filesystems
• /usr/tmp
– fast – subject to 14 day purge, not backed up – check quota with quota -s /usr/tmp (usually 75Gb and 6000 inodes)
• All PEs read and write the same file simultaneously
– Language I/O with FFIO library global layer – MPI I/O – Language I/O with FFIO library global layer and Cray extensions for
assign Examples
• read and write to file restart.file from all PEs by using the FFIO library global layer assign -F global:128:2 f:restart.file
• use the FFIO library bufa layer to improve performance for file opened on Fortran unit 10 assign -F bufa:128:2 u:10
• Examples above use an unblocked file structure, normal Fortran files are blocked. To read the file without the global or bufa layers you must use assign -s unblocked f:filename
additional flexibility
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
Cray FFIO library
• FFIO is a set of I/O layers tuned for different I/O characteristics
• $TMPDIR
– fast – purged at end of job or session – shares quota with /usr/tmp
• $HOME
– slower – permanent, backed up – check quota with quota (usually 2Gb and 3500 inodes)
• bufa is an asynchronous buffering layer • performs read-ahead, write-behind • specify buffer size with -F bufa:bs:nbufs where bs
is the buffer size in units of 4Kbyte blocks, and nbufs is the number of buffers • buffer space increases your applications memory requirements
changed by calling GLIO_GROUP_MPI(comm) • specify buffer size with -F global:bs:nbufs whБайду номын сангаасre
bs is the buffer size in units of 4Kbyte blocks, and nbufs is the number of buffers per PE
• bufa and global do not allow backspace, or skipping over a partially read record. You can allow this behavior by using the cos layer in addition to bufa or global, but then setpos doesn’t work. assign -s cos:128,bufa:128:2 f:filename
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
More on FFIO
• There are many other FFIO layers, some pretty obscure
– cache and cachea layers, good for random access files
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
Types of I/O
• Language I/O: Fortran or C (ANSI or POSIX) • Cray FFIO library (can be used from Fortran or C) • MPI I/O • Cray extensions to Fortran and C I/O (mostly for
– Language I/O – MPI I/O – Increase language I/O performance with FFIO library
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
I/O Strategies - Shared files
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
global FFIO layer
• global is a caching and buffering layer which enables multiple PEs to read and write to the same file
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
File positioning with the global FFIO layer
• Positioning of a read or write is your responsibility • File pointers are private • Fortran
ffwrite
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
The assign command
• the assign command controls
– controls which FFIO layer is active – striping across multiple partitions – lots more
• use the FFIO library bufa layer to improve performance for all unformatted sequential Fortran files assign -F bufa:128:2 g:su
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
• scope of assign
– File name – Fortran unit number – File type (e.g. all sequential unformatted files)
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
T3E Overview
• T3E is a set of Processing Elements (PE) connected by a fast 3D torus.
• PEs do not have local disk • All PEs access all filesystems equivalently • Path for I/O generally looks like:
compatibility with PVP systems)
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
I/O Strategies - Exclusive access files
• Each PE reads and writes to a separate file
• Buffering of data (configurable size) • Caching of data (configurable size) • Available to regular Fortran I/O without reprogramming • Available for C through POSIX-like calls, e.g. ffopen,
I/O Strategies - Communication and I/O PE
• One PE coordinates reading and writing and communicates data back and forth between other PEs via message passing
• if one PE has already read the data, an additional read request from another PE will result in a remote memory copy
• file open is a synchronizing event • By default, all PEs must open a global file, this can be
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
I/O Strategies for the T3E
Jonathan Carter NERSC User Services
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
– Use a direct access file, and read/write(rec=num) – Use Cray extensions setpos and getpos to position file pointer
(not portable)
•C
– Use ffseek
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
– Language I/O – MPI I/O – Increase language I/O performance with FFIO library (C must use
POSIX style calls)
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
assign Examples
• To see all active assigns assign -V
• To remove all active assigns assign -R
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
bufa FFIO layer
– user buffer space – system buffer space – I/O device buffer space
‹#›
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
Filesystems
• /usr/tmp
– fast – subject to 14 day purge, not backed up – check quota with quota -s /usr/tmp (usually 75Gb and 6000 inodes)
• All PEs read and write the same file simultaneously
– Language I/O with FFIO library global layer – MPI I/O – Language I/O with FFIO library global layer and Cray extensions for
assign Examples
• read and write to file restart.file from all PEs by using the FFIO library global layer assign -F global:128:2 f:restart.file
• use the FFIO library bufa layer to improve performance for file opened on Fortran unit 10 assign -F bufa:128:2 u:10