TestU01: Difference between revisions
No edit summary |
Citation bot (talk | contribs) Add: s2cid. | Use this bot. Report bugs. | #UCB_CommandLine |
||
| (34 intermediate revisions by 30 users not shown) | |||
| Line 1: | Line 1: | ||
{{Short description|A collection of utilities for empirical randomness testing}} |
|||
{{Wikify|date=October 2011}} |
|||
'''TestU01''' is a [[software library]], implemented in the [[ANSI C]] language, that offers a collection of utilities for the [[Empirical test|empirical]] [[randomness tests|randomness testing]] of [[random number generator]]s (RNGs).<ref name="TestU01">[http://simul.iro.umontreal.ca/testu01/tu01.html The TestU01 web site].</ref> The library was first introduced in 2007 by Pierre L’Ecuyer and Richard Simard of the [[Université de Montréal]].<ref name="Original Paper">Pierre L’Ecuyer & Richard Simard (2007), "[http://dl.acm.org/citation.cfm?doid=1268776.1268777 TestU01: A Software Library in ANSI C for Empirical Testing of Random Number Generators]", ''[[ACM Transactions on Mathematical Software]]'', 33: 22.</ref> |
|||
{{dead end|date=October 2011}} |
|||
| ⚫ | The library implements several types of random number generators, including some proposed in the literature and some found in widely used software. It provides general implementations of the classical statistical tests for random number generators, as well as several others proposed in the literature, and some original ones. These tests can be applied to the generators predefined in the library, user-defined generators, and streams of random numbers stored in files. Specific tests suites for either sequences of [[uniform distribution (continuous)|uniform]] random numbers in [0,1] or bit sequences are also available. Basic tools for plotting vectors of points produced by generators are provided as well. |
||
'''TestU01''' is a [[software library]], implemented in the [[ANSI C]] language, and offering a collection of utilities for the empirical statistical testing of uniform [[random number generator]]s.<ref name="TestU01">[http://www.iro.umontreal.ca/~simardr/testu01/tu01.html],The TestU01 Site.</ref> |
|||
| ⚫ | The library implements several types of random number generators |
||
== |
== History == |
||
| ⚫ | An initial battery of randomness tests for RNGs was suggested in the 1969 first edition of ''[[The Art of Computer Programming]]'' by [[Donald Knuth]]. Knuth's tests were then supplanted by [[George Marsaglia]]'s [[Diehard tests]] (1996) consisting of fifteen different tests. The inability to modify the test parameters or add new tests led to the development of the TestU01 library. |
||
TestU01, a software library implemented in the ANSI C language, and offering a collection of utilities for the empirical statistical testing of uniform random number generators (RNGs). |
|||
| ⚫ | |||
It provides general implementations of the classical statistical tests for RNGs, as well as several others tests proposed in the literature, and some original ones. Predefined tests suites for sequences of uniform random numbers over the interval (0, 1) and for bit sequences are available. Tools are also offered to perform systematic studies of the interaction between a specific test and the structure of the point sets produced by a given family of RNGs. That is, for a given kind of test and a given class of RNGs, to determine how large should be the sample size of the test, as a function of the generator's period length, before the generator starts to fail the test systematically. Finally, the library provides various types of generators implemented in generic form, as well as many specific generators proposed in the literature or found in widely used software. The tests can be applied to instances of the generators predefined in the library, or to user-defined generators, or to streams of random numbers produced by any kind of device or stored in files. |
|||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | When a specific test is applied to a sample of size ''n'' produced by an RNG, the [[p-value|''p''-value]] of the test usually will remain reasonable as the sample size increases until the sample size hits ''n''<sub>0</sub>, say. After that, the ''p''-value diverges to 0 or 1 with exponential speed. Module 4 allows the researcher to study the interaction between a specific test and the structure of the point sets produced by a given family of RNGs. This technique can be used to determine how large the sample size should be, as a function of the generator's period length, before the generator starts to fail the test systematically. |
||
| ⚫ | |||
| ⚫ | |||
TESTU01 offers several batteries of tests including "Small Crush" (which consists of 10 tests), "Crush" (96 tests), and "Big Crush" (106 tests). The specific tests applied by each battery are detailed in the user's guide.<ref name="User Guide">[http://www.iro.umontreal.ca/~simardr/testu01/guideshorttestu01.pdf TestU01 User's Guide].</ref> On a 1.7 GHz [[Pentium 4]] running [[Red Hat Linux]] 9.0, for a simple RNG, Small Crush takes about 2 minutes. Crush takes about 1.7 hours. Big Crush takes about 4 hours. For a more complex RNG, all these times |
|||
To use Marsaglia's program, the user creates a file of three million random numbers, and the program analyzes these numbers. There are some notable difficulties with DIEHARD. |
|||
| ⚫ | |||
First, it is not user-friendly. Second,it does not offer many tests—about 15. Third, the parameters of the tests are fixed, and it is often |
|||
advantageous to vary them—see the online appendix to this article at the JAE Data Archive for a demonstration of this point.<ref name="Demonstration @JAE Data Archive">[http://www.econ.queensu.ca/jae],Online JAE appendix demonstrating parameter variability advantage.</ref> Fourth, it is not extensible—new tests cannot be added. Fifth, while these tests might have been stringent 10 years ago, they are not now. |
|||
L’Ecuyer, long one of the world's top RNG researchers in general, and specifically one of the very best in testing of RNGs, is just the person to remedy these deficiencies and produce the successor to DIEHARD. He has done so with the programming assistance of Simard, and together they have created TESTU01. Just as many RNGs that passed the Knuth tests failed the DIEHARD tests, so many RNGs that pass the DIEHARD tests fail the TESTU01 tests. |
|||
== |
== Limitations == |
||
TestU01 only accepts 32-bit inputs, and interprets them as values in the range [0, 1]. This causes it to be more sensitive to flaws in the most-significant bits than the least significant bits. It is important to test general-purpose generators in bit-reversed form, to verify their suitability for applications which use the low-order bits.<ref name="vigna"> |
|||
| ⚫ | |||
{{cite journal |
|||
| ⚫ | |||
| last=Vigna | first=Sebastiano |
|||
| ⚫ | |||
| title=An experimental exploration of Marsaglia's xorshift generators, scrambled |
|||
| ⚫ | |||
| journal=ACM Transactions on Mathematical Software |
|||
| ⚫ | |||
| volume=42 | issue=4 | date=July 2016 | page=30<!--Actually article number--> |
|||
| doi=10.1145/2845077 |
|||
| arxiv=1402.6246 |
|||
| s2cid=13936073 |
|||
| url=http://vigna.di.unimi.it/ftp/papers/xorshift.pdf |
|||
}}</ref>{{Rp|4}} |
|||
Generators which produce 64 bits of output additionally require separate tests for their high and low halves.<ref>{{cite tech report |
|||
| ⚫ | When a specific test is applied to a sample of size n produced by an RNG, the p-value of the test usually will remain |
||
|title=PCG: A Family of Simple Fast Space-Efficient Statistically Good Algorithms for Random Number Generation |
|||
|first=Melissa E. |last=O'Neill |
|||
|publisher=[[Harvey Mudd College]] |
|||
|id=HMC-CS-2014-0905 |
|||
|date=5 September 2014 |
|||
|url=http://www.pcg-random.org/pdf/hmc-cs-2014-0905.pdf#page=53 |
|||
}}</ref>{{Rp|51}} |
|||
== See also == |
|||
As far as testing is concerned, most users who are not specialists in random numbers will not want to choose from among the many statistical tests available, set parameters for each one and apply them serially; i.e., most users will not be concerned with (ii) above. Rather, most users will want to use the per-programmed batteries of tests (iii) to test the RNGs that are used in their statistical packages. To satisfy the needs of such users, TESTU01 offers three batteries of tests called ‘Small Crush’ (which consists of 10 tests), ‘Crush’ (60 tests) and ‘Big Crush’ (45 tests).4 |
|||
* [[Randomness tests]] |
|||
The specific tests applied by each battery are given in the User Guide.5 On a 1.7 GHz Pentium 4 |
|||
* [[Diehard tests]] |
|||
running Red Hat Linux 9.0, for a simple RNG, Small Crush takes about 2 minutes. Crush takes about 1.7 hours. Big Crush takes about 12 hours. (For a more complex RNG, all these times |
|||
* [[PractRand]] |
|||
| ⚫ | |||
==References== |
==References== |
||
{{reflist}} |
|||
<references /> |
|||
[[Category:Computer libraries]] |
[[Category:Computer libraries]] |
||
[[Category:Random number generation]] |
|||
[[Category:Statistical software]] |
|||
Latest revision as of 09:40, 25 July 2023
TestU01 is a software library, implemented in the ANSI C language, that offers a collection of utilities for the empirical randomness testing of random number generators (RNGs).[1] The library was first introduced in 2007 by Pierre L’Ecuyer and Richard Simard of the Université de Montréal.[2]
The library implements several types of random number generators, including some proposed in the literature and some found in widely used software. It provides general implementations of the classical statistical tests for random number generators, as well as several others proposed in the literature, and some original ones. These tests can be applied to the generators predefined in the library, user-defined generators, and streams of random numbers stored in files. Specific tests suites for either sequences of uniform random numbers in [0,1] or bit sequences are also available. Basic tools for plotting vectors of points produced by generators are provided as well.
History
[edit]An initial battery of randomness tests for RNGs was suggested in the 1969 first edition of The Art of Computer Programming by Donald Knuth. Knuth's tests were then supplanted by George Marsaglia's Diehard tests (1996) consisting of fifteen different tests. The inability to modify the test parameters or add new tests led to the development of the TestU01 library.
Features
[edit]TestU01 offers four groups of modules for analyzing RNGs:
- Implementing (pre-programmed) RNGs;
- Implementing specific statistical tests;
- Implementing batteries of statistical tests;
- Applying tests to entire families of RNGs.
When a specific test is applied to a sample of size n produced by an RNG, the p-value of the test usually will remain reasonable as the sample size increases until the sample size hits n0, say. After that, the p-value diverges to 0 or 1 with exponential speed. Module 4 allows the researcher to study the interaction between a specific test and the structure of the point sets produced by a given family of RNGs. This technique can be used to determine how large the sample size should be, as a function of the generator's period length, before the generator starts to fail the test systematically.
TESTU01 offers several batteries of tests including "Small Crush" (which consists of 10 tests), "Crush" (96 tests), and "Big Crush" (106 tests). The specific tests applied by each battery are detailed in the user's guide.[3] On a 1.7 GHz Pentium 4 running Red Hat Linux 9.0, for a simple RNG, Small Crush takes about 2 minutes. Crush takes about 1.7 hours. Big Crush takes about 4 hours. For a more complex RNG, all these times increase by a factor of two or more. For comparison, the Diehard tests take about 15 seconds to run.
Limitations
[edit]TestU01 only accepts 32-bit inputs, and interprets them as values in the range [0, 1]. This causes it to be more sensitive to flaws in the most-significant bits than the least significant bits. It is important to test general-purpose generators in bit-reversed form, to verify their suitability for applications which use the low-order bits.[4]: 4
Generators which produce 64 bits of output additionally require separate tests for their high and low halves.[5]: 51
See also
[edit]References
[edit]- ^ The TestU01 web site.
- ^ Pierre L’Ecuyer & Richard Simard (2007), "TestU01: A Software Library in ANSI C for Empirical Testing of Random Number Generators", ACM Transactions on Mathematical Software, 33: 22.
- ^ TestU01 User's Guide.
- ^ Vigna, Sebastiano (July 2016). "An experimental exploration of Marsaglia's xorshift generators, scrambled" (PDF). ACM Transactions on Mathematical Software. 42 (4): 30. arXiv:1402.6246. doi:10.1145/2845077. S2CID 13936073.
- ^ O'Neill, Melissa E. (5 September 2014). PCG: A Family of Simple Fast Space-Efficient Statistically Good Algorithms for Random Number Generation (PDF) (Technical report). Harvey Mudd College. HMC-CS-2014-0905.