Language Syntax Timeseries & Extension Experimental Examples Articles Specifications Playground Home Contact

Specifications


LispTick Streaming Protocol

TODO


LispTick File Format

LispTick can use lots of different format as source (.csv, dataview export, InfluxDB…). But we have created our own archive file format which if both compact and very fast as it uses direct access.

All int values, int32, int64, int with variable number of bytes, are stored using Little Endian order.

File

+----------------+------------+-------------+------------------------------------------------+
|     Offset     | bytes Size |   Value     |         Description                            |
+----------------+------------+-------------+------------------------------------------------+
|              0 |          8 | 4c 69 73 70 | "LispTick" header                              |
|                |            | 54 69 63 6b |                                                |
+----------------+------------+-------------+------------------------------------------------+
|              8 |         S1 |     ...     | Data Block 1                                   |
+----------------+------------+-------------+------------------------------------------------+
|           S1+8 |         S2 |     ...     | Data Block 2                                   |
+----------------+------------+-------------+------------------------------------------------+
|           ...  |     ...    |     ...     | Data Blocks                                    |
+----------------+------------+-------------+------------------------------------------------+
| S1+...+Sn-1 +8 |         Sn |     ...     | Data Block n                                   |
+----------------+------------+-------------+------------------------------------------------+
|           Coff |  Joff-Coff |     ...     | Intrument(s) List Data Block                   |
+----------------+------------+-------------+------------------------------------------------+
|           Joff | FS-Joff-32 |     ...     | Json Scheme Data Block                         |
+----------------+------------+-------------+------------------------------------------------+
|            -32 |          8 |     Coff    | Offset of Intrument(s) List Data Block (int64) |
+----------------+------------+-------------+------------------------------------------------+
|            -24 |          4 |      Cs     | Number of Instrument(s) (int32)                |
+----------------+------------+-------------+------------------------------------------------+
|            -20 |          8 |     Joff    | Offest of Json Scheme Data Block (int64)       |
+----------------+------------+-------------+------------------------------------------------+
|            -12 |          4 |      Js     | Json Scheme size encoded (int32)               |
+----------------+------------+-------------+------------------------------------------------+
|             -8 |          8 | 6b 63 69 54 | "kciTpsiL" footer                              |
|                |            | 70 73 69 4c |                                                |
+----------------+------------+-------------+------------------------------------------------+

FS is file size in bytes.

Data Block

+--------+------------+--------------------------------------+
| Offset | bytes Size |           Description                |
+--------+------------+--------------------------------------+
| 0      |        1-4 | Data Block Header or Zstd            |
+--------+------------+--------------------------------------+
| 0 or 1 |    n bytes | data encoded following Header format |
+--------+------------+--------------------------------------+

If Data Block Header is 0x28 you must have those 4 bytes at start:

Which is Zstd Magic Number, this block must be directly read by a Ztd decompressor.

Data Block Header
+------------+------------------+---------------------------------------------------------+
| Bit Number | Field Name       |                       Description                       |
+------------+------------------+---------------------------------------------------------+
|    0-2     | Record Size      | number of bytes per record, 0 for 8 bytes, default      |
+------------+------------------+---------------------------------------------------------+
|    3-5     | Compression Mode | 0 for no compression, 1 for single value, 5 for Zstd    |
+------------+------------------+---------------------------------------------------------+
|     6      | Factor Flag      | if set, each record must be multiplied by a factor      |
+------------+------------------+---------------------------------------------------------+
|     7      | Delta Flag       | if set, each record must be computed using previous one |
+------------+------------------+---------------------------------------------------------+

Examples:

Intrument(s) List Data Block data

Once Data Block Header as been read, data is as following:

+------------+------------+------------------------------------------------------+
| bytes Size | Field Name |                    Description                       |
+------------+------------+------------------------------------------------------+
|          4 | I1Cs       | Instrument 1 Code Size, string size (int32)          |
+------------+------------+------------------------------------------------------+
|       I1Cs | I1Name     | Instrument 1 Code (no ending zero it's a Go string)  |
+------------+------------+------------------------------------------------------+
|          4 | I1Fs       | Instrument 1 Number of Fields (int32)                |
+------------+------------+------------------------------------------------------+
|          8 | I1Foff     | Offset of Instrument 1 Fields Description (int64)    |
+------------+------------+------------------------------------------------------+
+------------+------------+------------------------------------------------------+
|          4 | I2Cs       | Instrument 2 Code Size, string size (int32)          |
+------------+------------+------------------------------------------------------+
|       I2Cs | I2Name     | Instrument 2 Code (no ending zero it's a Go string)  |
+------------+------------+------------------------------------------------------+
|          4 | I2Fs       | Instrument 2 Number of Fields (Little Endian int32)  |
+------------+------------+------------------------------------------------------+
|          8 | I2Foff     | Offset of Instrument 2 Fields Description (int64)    |
+------------+------------+------------------------------------------------------+
+------------+------------+------------------------------------------------------+
|        ... | ...        | All instruments from 1 to Cs                         |
+------------+------------+------------------------------------------------------+
+------------+------------+------------------------------------------------------+
|          4 | ICsCs      | Instrument Cs Code Size, string size (int32)         |
+------------+------------+------------------------------------------------------+
|      ICsCs | ICsName    | Instrument Cs Code (no ending zero it's a Go string) |
+------------+------------+------------------------------------------------------+
|          4 | ICsFs      | Instrument Cs Number of Fields (int32)               |
+------------+------------+------------------------------------------------------+
|          8 | ICsFoff    | Offset of Instrument Cs Fields Description (int64)   |
+------------+------------+------------------------------------------------------+