File Formats

This section documents the structure of Aion binary data files. The files described in this section can be created using the functions:

and loaded into Aion using the functions:

File Header

All binary data files use the same header format described in Table 54.

Table 54 Data File Header Format

Offset

Meaning

Encoding

5 - 0

Magic Number

0x49 0x4E 0x45 0x42 0x49 0x4E (ASCII for “INEBIN”)

6

Reserved

0x00

7

Matrix Type

ASCII: “B”, “Z”, “R”, or “C” representing boolean, integer, real, and complex matrices respectively.

11 - 8

Number Rows

4-byte little endian unsigned integer

15 - 12

Number Columns

4-byte little endian unsigned integer

16 - xx

Matrix Data

The application specific matrix data

Boolean Matrix Data Files

Boolean matrices are stored in a packed format with one bit per matrix entry. Entries are written in reading order across rows and then by row.

The zero based bit index into the matrix data can be calculated by:

\[i _ { index } = N _ c \left ( r - 1 \right ) + c - 1\]

Where \(i _ { index }\) is the bit index into the matrix data associated with a given matrix entry, \(N _ c\) indicates the number of matrix columns, \(r\) represents the one based row number, and \(c\) represents the one based column number.

The byte offset into the file where a given entry is stored can then be determined by:

\[i _ { offset } = \left \lfloor \frac{ i _ { index }}{8} \right \rfloor + 16\]

The bit offset into the byte is given by:

\[i _ b = \text{Remainder Of } \frac{i _ { index }}{8}\]

Bit offset 0 represents the least sigificant bit into each byte, bit offset 7 represents the most significant bit into each byte.

The matrix

\[\begin{split}A _ { i,j } \in \mathbb{B} \;\;\;\;\;\; A = \begin{bmatrix} 1 & 0 & 0 & 1 & 1 \\ 0 & 0 & 1 & 1 & 0 \\ 0 & 0 & 0 & 1 & 0 \end{bmatrix}\end{split}\]

would create the following binary file described in Table 55.

Table 55 Example Boolean Binary File

Offset

Contents

Representing

Meaning

0

0x49

“INEBIN”

File magic number.

1

0x4E

2

0x45

3

0x42

4

0x49

5

0x4E

6

0x00

Reserved

7

0x42

“B”

Indicates a boolean matrix.

8

0x03

3

Number rows

9

0x00

10

0x00

11

0x00

12

0x05

5

Number columns

13

0x00

14

0x00

15

0x00

16

0x99

10011 001

Matrix Data

17

0x21

10 00010 0

Below is sample code that writes a boolean matrix:

bool writeBooleanMatrix(
        const char* filename,
        unsigned    numberRows,
        unsigned    numberColumns,
        const bool* matrix
    ) {
    bool success = true;

    std::ofstream f(filename, std::ios::binary);
    if (f) {
        unsigned      i      = 6;
        std::uint8_t* buffer = new std::uint8_t[4096];

        std::memcpy(buffer, "INEBIN", 6);
        buffer[i++] = 0;
        buffer[i++] = 'B';

        buffer[i++] = static_cast<std::uint8_t>(numberRows      );
        buffer[i++] = static_cast<std::uint8_t>(numberRows >>  8);
        buffer[i++] = static_cast<std::uint8_t>(numberRows >> 16);
        buffer[i++] = static_cast<std::uint8_t>(numberRows >> 24);

        buffer[i++] = static_cast<std::uint8_t>(numberColumns      );
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >>  8);
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >> 16);
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >> 24);

        std::uint8_t  mask     = 1;
        unsigned long rowIndex = 0;
        std::uint8_t  v        = 0;
        while (success && rowIndex < numberRows) {
            unsigned long columnIndex = 0;
            while (success && columnIndex < numberColumns) {
                if (matrix[numberColumns * rowIndex + columnIndex]) {
                    v |= mask;
                }

                mask <<= 1;
                if (mask == 0) {
                    buffer[i++] = v;
                    v = 0;

                    mask = 1;

                    if (i == 4096) {
                        f.write(reinterpret_cast<const char*>(buffer), 4096);
                        success = static_cast<bool>(f);
                        i       = 0;
                    }
                }

                ++columnIndex;
            }

            ++rowIndex;
        }

        if (success) {
            if (mask != 1 || i != 0) {
                if (mask != 1) {
                    buffer[i++] = v;
                }

                f.write(reinterpret_cast<const char*>(buffer), i);
                success = static_cast<bool>(f);
            }
        }

        if (success) {
            f.close();
            success = static_cast<bool>(f);
        } else {
            success = false;
        }

        delete[] buffer;
    } else {
        success = false;
    }

    return success;
}

Integer Matrix Data Files

Integer matrices are stores in reading order across rows and then by row. Each entry is stored in signed little-endian format with 8-bytes per entry.

The zero based byte offset into the file where a given row/column entry starts is given by:

\[i _ { index } = 8 \left [ N _ c \left ( r - 1 \right ) + c - 1 \right ] + 16\]

The matrix

\[\begin{split}A _ { i,j } \in \mathbb{Z} \;\;\;\;\;\; A = \begin{bmatrix} 1 & 65536 & 0x010203040506 \\ -1 & -65536 & - 2 ^ {62} \end{bmatrix}\end{split}\]

would create the following binary file described in Table 56.

Table 56 Example Integer Binary File

Offset

Contents

Representing

Meaning

0

0x49

“INEBIN”

File magic number.

1

0x4E

2

0x45

3

0x42

4

0x49

5

0x4E

6

0x00

Reserved

7

0x42

“Z”

Indicates an integer matrix.

8

0x02

2

Number rows

9

0x00

10

0x00

11

0x00

12

0x03

3

Number columns

13

0x00

14

0x00

15

0x00

16

0x01

1

Matrix Data

17

0x00

18

0x00

19

0x00

20

0x00

21

0x00

22

0x00

23

0x00

24

0x00

65536

25

0x00

26

0x01

27

0x00

28

0x00

29

0x00

30

0x00

31

0x00

32

0x08

0x0102030405060708

33

0x07

34

0x06

35

0x05

36

0x04

37

0x03

38

0x02

39

0x01

40

0xFF

-1

41

0xFF

42

0xFF

43

0xFF

44

0xFF

45

0xFF

46

0xFF

47

0xFF

48

0x00

-65536

49

0x00

50

0xFF

51

0xFF

52

0xFF

53

0xFF

54

0xFF

55

0xFF

48

0x00

\(-2^{62}\)

49

0x00

50

0x00

51

0x00

52

0x00

53

0x00

54

0x00

55

0xC0

The sample code below writes an integer matrix to a binary file.

bool writeIntegerMatrix(
        const char*      filename,
        unsigned         numberRows,
        unsigned         numberColumns,
        const long long* matrix
    ) {
    bool success = true;

    std::ofstream f(filename, std::ios::binary);
    if (f) {
        unsigned      i      = 6;
        std::uint8_t* buffer = new std::uint8_t[4096];

        std::memcpy(buffer, "INEBIN", 6);
        buffer[i++] = 0;
        buffer[i++] = 'Z';

        buffer[i++] = static_cast<std::uint8_t>(numberRows      );
        buffer[i++] = static_cast<std::uint8_t>(numberRows >>  8);
        buffer[i++] = static_cast<std::uint8_t>(numberRows >> 16);
        buffer[i++] = static_cast<std::uint8_t>(numberRows >> 24);

        buffer[i++] = static_cast<std::uint8_t>(numberColumns      );
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >>  8);
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >> 16);
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >> 24);

        unsigned long rowIndex = 0;
        while (success && rowIndex < numberRows) {
            unsigned long columnIndex = 0;
            while (columnIndex < numberColumns) {
                long long v = matrix[numberColumns * rowIndex + columnIndex];
                *reinterpret_cast<std::int64_t*>(buffer + i) = v;
                i += sizeof(std::int64_t);

                if (i >= 4096) {
                    f.write(reinterpret_cast<const char*>(buffer), 4096);
                    success = static_cast<bool>(f);
                    i       = 0;
                }

                ++columnIndex;
            }

            ++rowIndex;
        }

        if (i != 0) {
            f.write(reinterpret_cast<const char*>(buffer), i);
            success = static_cast<bool>(f);
        }

        if (success) {
            f.close();
            success = static_cast<bool>(f);
        }

        delete[] buffer;
    } else {
        success = false;
    }

    return success;
}

Real Matrix Data Files

Real matrices are stores in reading order across rows and then by row. Each entry is stored as double precision values conforming with the IEEE 754-2008 standard as shown in Figure 224.

../_images/real_number_format.png

Figure 224 Real Number Format

Where the number value is represented by

\[(-1) ^ { sign } \left ( 1 + mantissa \right ) \times 2 ^ { exponent - 1023 }\]

The zero based byte offset into the file where a given row/column entry starts is given by:

\[i _ { index } = 8 \left [ N _ c \left ( r - 1 \right ) + c - 1 \right ] + 16\]

The matrix

\[\begin{split}A _ { i,j } \in \mathbb{R} \;\;\;\;\;\; A = \begin{bmatrix} 1 & 1.5 & 65536 \\ -1 & 0.375 & 0.0002 \end{bmatrix}\end{split}\]

would create the following binary file described in Table 57.

Table 57 Example Real Binary File

Offset

Contents

Representing

Meaning

0

0x49

“INEBIN”

File magic number.

1

0x4E

2

0x45

3

0x42

4

0x49

5

0x4E

6

0x00

Reserved

7

0x42

“R”

Indicates a real matrix.

8

0x02

2

Number rows

9

0x00

10

0x00

11

0x00

12

0x03

3

Number columns

13

0x00

14

0x00

15

0x00

16

0x00

1

Matrix Data

17

0x00

18

0x00

19

0x00

20

0x00

21

0x00

22

0xF0

23

0x3F

24

0x00

1.5

25

0x00

26

0x00

27

0x00

28

0x00

29

0x00

30

0xF8

31

0x3F

32

0x00

65536

33

0x00

34

0x00

35

0x00

36

0x00

37

0x00

38

0xF0

39

0x40

40

0x00

-1

41

0x00

42

0x00

43

0x00

44

0x00

45

0x00

46

0xF0

47

0xBF

48

0x00

0.375

49

0x00

50

0x00

51

0x00

52

0x00

53

0x00

54

0xD8

55

0x3F

56

0x2D

0.0002

57

0x43

58

0x1C

59

0xEB

60

0xE2

61

0x36

62

0x2A

63

0x3F

The sample code below writes a real matrix to a binary file.

bool writeRealMatrix(
        const char*   filename,
        unsigned      numberRows,
        unsigned      numberColumns,
        const double* matrix
    ) {
    bool success = true;

    std::ofstream f(filename, std::ios::binary);
    if (f) {
        unsigned      i      = 6;
        std::uint8_t* buffer = new std::uint8_t[4096];

        std::memcpy(buffer, "INEBIN", 6);
        buffer[i++] = 0;
        buffer[i++] = 'R';

        buffer[i++] = static_cast<std::uint8_t>(numberRows      );
        buffer[i++] = static_cast<std::uint8_t>(numberRows >>  8);
        buffer[i++] = static_cast<std::uint8_t>(numberRows >> 16);
        buffer[i++] = static_cast<std::uint8_t>(numberRows >> 24);

        buffer[i++] = static_cast<std::uint8_t>(numberColumns      );
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >>  8);
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >> 16);
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >> 24);

        unsigned long rowIndex = 0;
        while (success && rowIndex < numberRows) {
            unsigned long columnIndex = 0;
            while (columnIndex < numberColumns) {
                double v = matrix[numberColumns * rowIndex + columnIndex];
                *reinterpret_cast<double*>(buffer + i) = v;
                i += sizeof(double);

                if (i >= 4096) {
                    f.write(reinterpret_cast<const char*>(buffer), 4096);
                    success = static_cast<bool>(f);
                    i       = 0;
                }

                ++columnIndex;
            }

            ++rowIndex;
        }

        if (i != 0) {
            f.write(reinterpret_cast<const char*>(buffer), i);
            success = static_cast<bool>(f);
        }

        if (success) {
            f.close();
            success = static_cast<bool>(f);
        }

        delete[] buffer;
    } else {
        success = false;
    }

    return success;
}

Complex Matrix Data Files

Complex matrices are stores in reading order across rows and then by row. Each entry is stored as a pair of double precision values conforming with the IEEE 754-2008 standard as shown in Figure 224.

Real values are placed first in the file, followed by complex values.

The zero based byte offset into the file where a given row/column entry starts is given by:

\[i _ { index } = 16 \left [ N _ c \left ( r - 1 \right ) + c - 1 \right ] + 16\]

The matrix

\[\begin{split}A _ { i,j } \in \mathbb{R} \;\;\;\;\;\; A = \begin{bmatrix} 1+1,5i & 0.375+1.75i & 3 + 5i \\ 6+7i & 2 & 0.9375+31i \end{bmatrix}\end{split}\]

would create the following binary file described in Table 58.

Table 58 Example Complex Binary File

Offset

Contents

Representing

Meaning

0

0x49

“INEBIN”

File magic number.

1

0x4E

2

0x45

3

0x42

4

0x49

5

0x4E

6

0x00

Reserved

7

0x42

“C”

Indicates a complex matrix.

8

0x02

2

Number rows

9

0x00

10

0x00

11

0x00

12

0x03

3

Number columns

13

0x00

14

0x00

15

0x00

16

0x00

1

Matrix Data

17

0x00

18

0x00

19

0x00

20

0x00

21

0x00

22

0xF0

23

0x3F

24

0x00

1.5

25

0x00

26

0x00

27

0x00

28

0x00

29

0x00

30

0xF8

31

0x3F

32

0x00

0.375

33

0x00

34

0x00

35

0x00

36

0x00

37

0x00

38

0xD8

39

0x3F

40

0x00

1.75

41

0x00

42

0x00

43

0x00

44

0x00

45

0x00

46

0xFC

47

0x3F

48

0x00

3

49

0x00

50

0x00

51

0x00

52

0x00

53

0x00

54

0x08

55

0x40

56

0x00

5

57

0x00

58

0x00

59

0x00

60

0x00

61

0x00

62

0x14

63

0x40

64

0x00

6

65

0x00

66

0x00

67

0x00

68

0x00

69

0x00

70

0x18

71

0x40

72

0x00

7

73

0x00

74

0x00

75

0x00

76

0x00

77

0x00

78

0x1C

79

0x40

80

0x00

2

81

0x00

82

0x00

83

0x00

84

0x00

85

0x00

86

0x00

87

0x40

88

0x00

0

89

0x00

90

0x00

91

0x00

92

0x00

93

0x00

94

0x00

95

0x00

96

0x00

0.9375

97

0x00

98

0x00

99

0x00

100

0x00

101

0x00

102

0xEE

103

0x3F

104

0x00

31

105

0x00

106

0x00

107

0x00

108

0x00

109

0x00

110

0x3F

111

0x40

The sample code below writes a complex matrix to a binary file.

bool writeComplexMatrix(
        const char*                 filename,
        unsigned                    numberRows,
        unsigned                    numberColumns,
        const std::complex<double>* matrix
    ) {
    bool success = true;

    std::ofstream f(filename, std::ios::binary);
    if (f) {
        unsigned      i      = 6;
        std::uint8_t* buffer = new std::uint8_t[4096];

        std::memcpy(buffer, "INEBIN", 6);
        buffer[i++] = 0;
        buffer[i++] = 'C';

        buffer[i++] = static_cast<std::uint8_t>(numberRows      );
        buffer[i++] = static_cast<std::uint8_t>(numberRows >>  8);
        buffer[i++] = static_cast<std::uint8_t>(numberRows >> 16);
        buffer[i++] = static_cast<std::uint8_t>(numberRows >> 24);

        buffer[i++] = static_cast<std::uint8_t>(numberColumns      );
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >>  8);
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >> 16);
        buffer[i++] = static_cast<std::uint8_t>(numberColumns >> 24);

        unsigned long rowIndex = 0;
        while (success && rowIndex < numberRows) {
            unsigned long columnIndex = 0;
            while (columnIndex < numberColumns) {
                const std::complex<double>& v = matrix[
                    numberColumns * rowIndex + columnIndex
                ];

                *reinterpret_cast<double*>(buffer + i) = v.real();
                i += sizeof(double);
                *reinterpret_cast<double*>(buffer + i) = v.imag();
                i += sizeof(double);

                if (i >= 4096) {
                    f.write(reinterpret_cast<const char*>(buffer), 4096);
                    success = static_cast<bool>(f);
                    i       = 0;
                }

                ++columnIndex;
            }

            ++rowIndex;
        }

        if (i != 0) {
            f.write(reinterpret_cast<const char*>(buffer), i);
            success = static_cast<bool>(f);
        }

        if (success) {
            f.close();
            success = static_cast<bool>(f);
        }

        delete[] buffer;
    } else {
        success = false;
    }

    return success;
}