Lisptick wiki Timeserie

Tick Time Series dedicated functions

backward

Shift all values to n previous time. n first updates are lost if keep is false or first time is used if keep is true.

(backward ts {n {keep}})

parameter

ts: input timeserie
n: optional (default is 1), number of shift or duration
keep: optional (default is false)

clockref aka cf

Define the timeserie as a clock reference, or metronome. For any operation between several timeseries there will be one update for output if there is one update for the clockref serie. This synchronization mode ensure that length of output timeserie is <= to clockref timeserie. Adding a ref-value ensure length of output timeserie is = to clockref timeserie, uncomputed values due to missing data from other series will be replaced by ref-value.

(cf ts {ref-value})       ;short keywork
(clockref ts {ref-value}) ;long keyword

parameter

ts: timeserie than will become the reference for time
ref-value: optional value used before any value the timeserie

delta

Returns timeserie where each point is the current value minus previous value of the input timeserie. So result length is input length-1, and first output point time is time of second input point time.

(delta ts)

parameter

ts: input timeserie

fields

Return all available fields for a given source, code and date.

(fields source code date)

parameter

source: data provider source name
code: a code related to source (e.g. a RIC like “BNPP.PA”)
date: a LispTick Date without time

forward

Shift all values to n next time. n last updates are lost if keep is false or last time is used if keep is true.

(forward ts {n {keep}})

parameter

ts: input timeserie
n: optional (default is 1), number of shift or duration
keep: optional (default is false)

graphsample

Subsample timeserie in order to minimize number of points but to keep same result on screen. This is useful to get a smaller timeserie that will be drawn like the full timeserie if your output screen is nbr pixel width. As graphsample process on a stream it can’t know the future so sends lots of points at the beginning then less and less.

(graphsample nbr ts)

parameters

nbr: should be the number of pixels of the graph you will create with timeserie.
ts: input timeserie

example

Show all S&P E-Mini future bid prices not null on a Full HD screen (more than 7 million points)

(graphsample 1920 (timeserie @bid-price "refinitiv" "ESc1" 2016-01-20))

hist

Return histogram of data as a hash map. If argument is a simple timeserie, it returns the distribution of each value. If argument is a timeserie of array, it returns the sum of last element of array and use all previous element as the key, see sync for an example.

MEMORY WARNING hist result can be huge if there is a lot of different values.

(hist arg1 {arg2 {arg3 ...}}})

parameter

args: something

example

Show number of BTC trades per price. Here price is rounded to closer multiple of 10 to have less beans. We can see there are 1183 trades between 1455 and 1464 (1460 bean). See sync for a turn-over histogram example. See also mean, median, sd for other statistical functions.

(def prices
  (timeserie @trade-price "bitstamp" "BTC" 2018-02-28))
(hist (round prices 1))

history

Return all available dates for a given source and code.

As history can be huge depending on source better call through a LispTick client with streaming (Walk method).

(history source code)

source: data provider source name
code: a code related to source (e.g. a RIC like “BNPP.PA”)

keep

Only keep points in timeserie that fulfill the given condition

(keep ts func args2 args3 ...)

parameters

ts: input timeserie
func: can be <, >, <=, >=, =, not= or any function returning a boolean, each ts point is used as 1st argument. Can also be a timeserie of boolean, ts points are kept if func is true at the same time.
args: argument of func after timeserie point

examples

Change timeserie ts to keep only positif values

(keep ts > 0)

Limit perimeter to code ending with “.PA”

(keep (perimeter "refinitiv" 2016-02-01) has-suffix ".PA")

Only keep trade prices when trade volume is greater that 2

[
  (label "all trade prices"
    (timeserie @trade-price "bitstamp" "BTC" 2018-02-28))
  (label "trade prices with volume >2"
    (keep
      (timeserie @trade-price "bitstamp" "BTC" 2018-02-28)
      (> (timeserie @trade-volume "bitstamp" "BTC" 2018-02-28) 2)))
]

Kurtosis

Return unbiased kurtosis of samples. Kurtosis (from Greek: κυρτός, kyrtos or kurtos, meaning “curved, arching”) is a measure of the “tailedness” of the probability distribution of a real-valued random variable. Like skewness, kurtosis describes the shape of a probability distribution. If argument is a timeserie, it returns online kurtosis at any time.

(kurtosis arg1 {arg2 {arg3 ...}}})

parameter

args: something (value, array, timeserie), trick: use a predefined hist for fast computation

example

See also hist, mean, median for other statistical functions.

Show trade price Kurtosis for BTC on 28th february 2018.

(kurtosis (timeserie @trade-price "bitstamp" "BTC" 2018-02-28))

Show kurtosis for BTC 1 minute log-return on 28th february 2018.

(defn log-return[ts lag]
  (ln
    (+ 1
      (/ (- ts (+ ts lag)) (+ ts lag)))))
(kurtosis
  (log-return
    (timeserie @trade-price "bitstamp" "BTC" 2018-02-28)
    1m))

label

Give a name to the timeserie that can be used in a graph for example.

(label name ts)

parameters

name: a string, a function (name will used)
ts: input timeserie

localize

Set time location, useful to ensure time precision when created manually.

(localize location t|ts )

parameters

location: timezone, can be any tz data name like “US/Eastern” see TZ Wiki
t|ts: input time or timeserie

examples

Tests ensuring daylight saving is well handled

(assert (=
  (localize "Europe/Paris" 2017-12-18T14:12)
  (localize "US/Eastern" 2017-12-18T08:12)))
(assert (=
  (localize "Europe/Paris" 2017-11-01T14:12)
  (localize "US/Eastern" 2017-11-01T09:12)))

map

Apply a function to each element of a serie. Serie can be list, array, hash(histogram) or a timeserie.

(map fn arg)

parameter

fn: a function applied to each element
arg: timeserie, array, histogram…

examples

Normalisation of an histogram, sum of all bean will be 1, price rounded to the nearest hundred (10 ^ 2). See hist for the histogram.

(def prices
  (timeserie @trade-price "bitstamp" "BTC" 2018-02-28))
(def histogram (hist (round prices 2)))
;compute all beans summation
(def total (last (+ histogram)))
;normalize with anonymous function 
(map (fn[a] (/ a total)) histogram)

map-reduce

Apply function with all given argument combinations using all available ressources.

(map-reduce func reduce args1 args2 ...)

parameter

func: function name called with each argument tuples (args1[0] args2[0]…) (args1[0] args2[1] …) …
reduce: how to choose point to keep, can be open (first), high (max), low (min), close (last), sum (+), count (len), all…
args1: a value, an array of values, a timeserie or a range used as an iterator
args2: a value, an array of values, a timeserie or a range used as an iterator
…

example

Compute number of trades for Bitcoin, Ether, Ripple from source bitstamp in February 2018. Show only the result value of timeserie, not the time (which is time of last trade).

(defn nbr-trade[d code]
  (subsample 1D count
    (timeserie @trade-price "bitstamp" code d)))
(vget
  (map-reduce nbr-trade sum
    (range 2018-02-01 2018-02-28 1D)
    ["BTC" "ETH" "XRP"]))

map-reduce-arg

As map-reduce but result is a timeserie of pairs like ([args1 args2 …] . result) instead of a simple timeserie of results.

(map-reduce-arg func reduce args1 args2 ...)

parameter

func: function name called with each argument tuples (args1[0] args2[0]…) (args1[0] args2[1] …) …
reduce: how to choose point to keep, can be open (first), high (max), low (min), close (last), sum (+), count (len), all…
args1: a value, an array of values, a timeserie or a range used as an iterator
args2: a value, an array of values, a timeserie or a range used as an iterator …

max

Max returns maximum of several values. If an argument is a timeserie, latest value is used for comparaison. If argument is a single timeserie, result is timeserie of maximum value at any point in time. If argument is a single tensor, result is maximum value in tensor. If max-arg is used, result is a pair with head (get result 0) as value and tail (get result 1) as position.

(max a {b} ...)

parameters

a: first input timeserie or value or tensor
b: second input timeserie or value
…: as many timeseries or values as you want

example

Use maximum of current and previous bid price to filter spikes.

(max
  (timeserie @bid-price "bitstamp" "BTC" 2018-02-14)
  (forward (timeserie @bid-price "bitstamp" "BTC" 2018-02-14)))

Get higer trade price value over a period.

(vget (last (max
  (timeserie @trade-price "bitstamp" "BTC" 2018-02-01 2018-02-28))))

mean

Return arithmetic mean of samples. If argument is a timeserie, it returns online mean at any time.

(mean arg1 {arg2 {arg3 ...}}})

parameter

args: something, trick: use a predefined hist for fast computation

example

See also hist, median, sd for other statistical functions.

Show mean trade price for BTC on 28th february 2018.

(mean (timeserie @trade-price "bitstamp" "BTC" 2018-02-28))

Show mean of amount in $ exchanged per trade for BTC on 28th february 2018.

(vget (last (mean
  (*
    (timeserie @trade-price "bitstamp" "BTC" 2018-02-28)
    (timeserie @trade-volume "bitstamp" "BTC" 2018-02-28)))))

median

Return median of samples. The median is the value separating the higher half from the lower half of a data sample. If argument is a timeserie, it returns median of all timeserie values.

MEMORY WARNING median memory usage can be huge if there is a lot of different values.

(median arg1 {arg2 {arg3 ...}}})

parameter

args: something, trick: use a predefined hist for fast computation

example

See also hist, mean, sd for other statistical functions.

Show median trade price for BTC on 28th february 2018.

(median (timeserie @trade-price "bitstamp" "BTC" 2018-02-28))

Show median amount in $ exchanged per trade for BTC on 28th february 2018.

(vget (last (median
  (*
    (timeserie @trade-price "bitstamp" "BTC" 2018-02-28)
    (timeserie @trade-volume "bitstamp" "BTC" 2018-02-28)))))

merge

Merge several timeseries into one, synchronize points from all timeseries.

(merge ts1 ts2 ...)

parameters

ts1: first input timeserie
ts2: second input timeserie
…: as many timeseries as you want

min

Min returns minimum of several values. If an argument is a timeserie latest value is used for comparaison. If argument is a single timeserie, result is timeserie of minimum value at any point in time. If argument is a single tensor, result is minimum value in tensor. If min-arg is used, result is a pair with head (get result 0) as value and tail (get result 1) as position.

(min a {b} ...)

parameters

a: first input timeserie or value or tensor
b: second input timeserie or value
…: as many timeseries or values as you want

See max for some examples

now

Retrieve current time

(now)

example

trade, bid & ask prices from previous hour until next hour for Bitcoin from Bitstamp. You can also see in this example how to define a function and how to label a timeserie with a string to have a nicer name on screen.

(defn ts[field]
  (label (str field)
  (timeserie field "bitstamp" "BTC" (- (now) 1h) (+ (now) 1h))))
[
  (ts @trade-price)
  (ts @bid-price)
  (ts @ask-price)
]

one

Replace value of every update by 1.0 .

(one ts)

parameter

ts: input timeserie

perimeter

Return all available codes for a given source and date or range.

As perimeter can be huge depending on source better call through a LispTick client with streaming (Walk method).

(perimeter source date {stop})

parameter

source: data provider source name
code: a code related to source (e.g. a RIC like “BNPP.PA”)
date: a LispTick Date without time, can be omitted depending on source
stop: a LispTick Date without time for end of range, included (optional)

prune

Filter points in timeserie where value doesn’t change.

(prune ts)

parameter

ts: input timeserie

range

Create an interator on an range of values. Should be used as an argument for map-reduce.

(range start stop step)

parameter

start: Starting value, included
stop: Ending value, included
step: Value to add at each step

example

Every day from 19 January 2018 until 1st May 2018, included

(range 2018-01-19 2018-05-01 1D)

reverse

Reverse timeserie timeline in order to apply value in past. This feature is dedicated to factor/spread adjustement that needs to be applied for all point in the past starting at a specific time.

!!! Be very carefull when using this feature as ALL timeserie points will be stored in memory!!!

(reverse ts {op})

parameters

ts: input timeserie
op: an optional operation to combine point in past, typically use + for spreads and * for factors

rts (recursive timeserie)

Timeserie of rescursive summation. Allows exponential moving average. Formula is: rts(t) = at * ts(t) + at-1 * rts(t-1)

(rts ts at at-1...)

parameters

ts: input timeserie
at: factor to apply to ts(t)
at-1: factor to apply to rst(t-1)

example

Compute full volume traded during the day at any time, equivalent to sigma function.

(rts (timeserie @trade-volume "bitstamp" "BTC" 2018-02-28) 1 1)

sd

Return unbiased standard deviation of samples. Standard deviation is a measure of the amount of variation or dispersion of a set of values. If argument is a timeserie, it returns online standard deviation at any time.

(sd arg1 {arg2 {arg3 ...}}})

parameter

args: something (value, array, timeserie), trick: use a predefined hist for fast computation

example

See also hist, mean, median for other statistical functions.

Show standard deviation trade price for BTC on 28th february 2018.

(sd (timeserie @trade-price "bitstamp" "BTC" 2018-02-28))

Show standard deviation amount in $ exchanged per trade for BTC on 28th february 2018.

(sd
  (*
    (timeserie @trade-price "bitstamp" "BTC" 2018-02-28)
    (timeserie @trade-volume "bitstamp" "BTC" 2018-02-28)))

sigma (+ with only one timeserie)

Returns a timeserie where each point is the summation of all previous points in the input timeserie.

(+ ts)

parameter

ts: input timeserie

example

Compute full volume traded during the day at any time.

(+ (timeserie @trade-volume "bitstamp" "BTC" 2018-02-28))

sign

Replace value of every update by its sign:

value < 0 → -1
value == 0 → 0
value > 0 → 1

(sign ts)

parameter

ts: input timeserie

skewness

Return unbiased skewness(asymmetry) of samples. Skewness is a measure of the asymmetry of the probability distribution of a real-valued random variable about its mean. The skewness value can be positive, zero, negative, or undefined. If argument is a timeserie, it returns online skewness at any time.

(skewness arg1 {arg2 {arg3 ...}}})

parameter

args: something (value, array, timeserie), trick: use a predefined hist for fast computation

example

See also hist, mean, median for other statistical functions.

Show trade price skewness for BTC on 28th february 2018.

(skewness (timeserie @trade-price "bitstamp" "BTC" 2018-02-28))

Show skewness for BTC 1 minute log-return on 28th february 2018.

(defn log-return[ts lag]
  (ln
    (+ 1
      (/ (- ts (+ ts lag)) (+ ts lag)))))
(vget (last (skewness
  (log-return
    (timeserie @trade-price "bitstamp" "BTC" 2018-02-28)
    1m))))

slice

Only keep timeserie points, tensor dimension in [start, stop[. If no timezone is given with start or stop it will be “local” to timeserie timezone. When start and stop are integers they represent absolute position, starting at 0, and last index to keep. No stop means keep everything after start. A negative stop of -n means forget n last point(s).

(slice x [start {stop {step}}]  {[start {stop {step}}]} ...)

parameters

x: input timeserie or array or list
start: a position (positive integer, 0 for first) or a time with or without date
stop: optional a position (integer, optional or 0 keeps all) or a time with or without date
step: optional integer to range start to stop step by step
as many [start {stop {step}}] as desired but no more than number of tensor dimensions

sliding

Replace each update by the specified value from the sliding window. Typical usage is a moving average.

(sliding period|int sampling ts)

parameters

period|int: a duration or an int, defines the window size in time or in number of updates
sampling: how to choose point to keep, can be open (first), high (max), low (min), close (last), sum (+), count (len), all…
ts: input timeserie

examples

Average weighted by number of updates, like Bollinger Band base. Here the average is done on 1000 updates or on a 2h window. Using a period instead of a number of updates allows to extend Bollinger bands to high frequency data.

Use graphsample to get same graph but with less data transmission.

(defn bollinger[d ts]
  (label (str "bollinger " d)
    (/
      (sliding d + ts)
      (sliding d len ts))))
[
(graphsample 1920 (timeserie @trade-price "bitstamp" "BTC" 2018-02-23))
(graphsample 1920 (bollinger 1000 (timeserie @trade-price "bitstamp" "BTC" 2018-02-23)))
(graphsample 1920 (bollinger 2h (timeserie @trade-price "bitstamp" "BTC" 2018-02-23)))
]

str

Applied on timseries, str creates a timeserie of string. If one input timeserie is a clockref, there will be exactly the same number of update as this timeserie.

(str arg1 {arg2 {arg3...}})

parameter

arg1: argument
arg2: 2nd argument (optional)
arg3: 3rd argument (optional) … as many optional arguments as you need

example

Show computation as nice timeserie of strings.

(def ts1
  (timeserie 2020-02-01 1 2020-02-03 5))
(def ts2
  (timeserie 2020-02-01 3 2020-02-02 2))
(str ts1 "+" ts2 "=" (clockref (+ ts1 ts2)))

subsample

Keep only one point per period from input timeserie, sampling defines which point to keep. Each value is taken in the range [i * period, (i+1) * period[, each time value is the time of original point.

(subsample period sampling ts)

parameters

period: a duration, defines the period size
sampling: how to choose point to keep, can be open (first), high (max), low (min), close (last), sum (+), count (len), kurtosis, mean, median, sd, skewness, all or an integer indicating the position to keep i.e. 1 for first 2 for second…
ts: input timeserie

examples

Request 10 minutes timebar for a stock.

Open High Low Close

(def 
  ts (timeserie @trade-price "bitstamp" "BTC" 2021-02-09)
  w 10m)
[
  [
    (subsample w open ts)
    (subsample w high ts)
    (subsample w low ts)
    (subsample w close ts)
  ]
]

Volume

(subsample 10m + (timeserie @trade-volume "bitstamp" "BTC" 2021-02-09))

Number of 1st limit bid update per hour for S&P E-Mini future

(subsample 1h len (timeserie @bid-price "refinitiv" "ESc1" 2016-01-20))

sync

Synchronise several timeseries and generate a timeserie of array. Use this fonction to create an export (csv for example) of synchronized series. If no value is available from a timeserie, previous or empty value is used. If one input timeserie is a clockref, there will be exactly the same number of update as this timeserie.

(sync ts1 {ts2 {ts3...}})

parameter

ts1: input timeserie
ts2: 2nd input timeserie (optional)
ts3: 3rd input timeserie (optional) … as many optional timeseries as you need

example

Show turnovers histogram in $ of BTC per price. Here price is rounded to closer multiple of 10 to have less beans. We can see there is a 4.5M$ turnover between 1495 and 1504 (1500 bean). See hist for a simpler histogram example.

(def prices
  (timeserie @trade-price "bitstamp" "BTC" 2018-02-28))
(def turnovers
  (*
   (timeserie @trade-price "bitstamp" "BTC" 2018-02-28)
   (timeserie @trade-volume "bitstamp" "BTC" 2018-02-28)))
(hist (sync (round prices 1) turnovers))

temporal

Creates a timeserie of temporal vector from a timeserie. From a timeserie of scalar it will keep the n latest updates into a vector of size n. Typical usage is to transform a simple timeserie into an input for a Temporal Convolutional Network.

You can use a duration as a filter to ensure all points are separated by the desired amount of time. If duration between two points is different from d, full matrice will be forgotten.

(temporal ts n {d})

parameter

ts: input timeserie
n: number of updates to keep, i.e. n latest upates
d: a duration (optional)

tget

Get time at a precise index in a timeserie. If index is a time return same time if an update exists or return time of update just before.

Result is “()” so isnull? true, when index unfound in serie.

(tget ts {i|time} )

parameters

ts: input timeserie
i: index or time of the data to retrieve, 0(default) is first -1 is last

example

Use the first and last timeserie times to compute the day trading duration. (tget ts) is equivalent to (tget ts 0) and (tget ts -1) is equivalent to (tget (last ts)).

(def ts (timeserie @"Turnover" "refinitiv" "BNPP.PA" 2020-03-02))
(- (tget ts -1) (tget ts))

time-as-array

Return times of timeserie as an array of datetime.

(time-as-array ts)

parameter

ts: input timeserie

time-as-value

Return timeserie of times instead of values.

(time-as-value ts)

parameter

ts: input timeserie

time-truncate

Return the result of rounding t or each timeserie t down to a multiple of d (since the zero time).

(time-truncate d t|ts)

parameter

d: a duration
t|ts: input time or timeserie

timeserie

Create timeserie by hand or retrieve data from a source

(timeserie @field "source" "code"|code-fn|array start {stop} {filter} )
(timeserie date value date value ...)
(timeserie (range ...) date-fn|value)

parameters

field: which tick data you want to look at, can be, depending on source:
- @“fieldname” field name as a string, depending on source
- @[trade|bid|ask]-[price|volume] for financial sources
- @[ask|bid]-[price|volume]-n retrieve nth limit (i.e. ask-price-1 should be equal to ask-price depending on source)
source: data provider source name
code|code-fn|array: object identification related to source
- code is generally a string (e.g. a RIC like “BNPP.PA”)
- a function returning a code from a date
- for tensor sources an array of numbers describing dimension bounds
- [] full dimension tensor
- [[42 52][-6 10]] latitude between 42°N and 52°N, longitude between -6°E and 10°E which is France for a geo source
- [44.8638098 -0.6684126] only one point from the tensor, so it becomes a scalar, this example is Bordeaux for a geo source
start: a LispTick date, if no time or location specified start is midnight (source location)
stop: optional LispTick Date, several days timeserie from [start to stop], if no time or location specified stop just before midnight next day (source location)
filter: optional string argument, used by source to filter output
range: a time/date range
date-fn: a function with a date as argument returning a value used for timeserie
value: value used for timeserie

examples

bid prices for Bitcoin from Bitstamp source, spikes come from lack of liquidity

(timeserie @bid-price "bitstamp" "BTC" 2018-02-14)

Hard coded time serie

(timeserie 2017-10-26T09:19 48.6 2017-10-26T10:30 49 2017-10-26T11:51 49.27)

a simple timeserie of day of the month

(timeserie (range 2019-01-01 2019-05-28 1D) day)

uniq

Filters out repeated values, each value appears only once in result. If argument is a timeserie, it returns a timeserie with the first occurrence of each value. If argument is an array, it returns an array with each value present only once. For all other argument type uniq is equivalent to identity.

MEMORY WARNING uniq memory usage can be huge if there is a lot of different values.

(uniq arg)

parameter

arg: something

example

Show first BTC trade price occurrence. Here price is rounded to closer multiple of 10 to have less different values.

(def prices
  (timeserie @trade-price "bitstamp" "BTC" 2018-02-28))
[
  (round prices 1)
  (label "uniq" (uniq (round prices 1)))
]

value-as-array

Return values of timeserie as an array of floats.

(value-as-array ts)

parameter

ts: input timeserie

version

Return LispTick version as a string.

(version)

vget

Get value at a precise index in a timeserie, works like tget. If index is a time return value at time if an update exists or just before.

Result is “()” so isnull? true, when index unfound in serie.

(vget ts {i|time} )

parameters

ts: input timeserie
i: index or time of the data to retrieve, 0(default) is first -1 is last

example

Use first timeserie value to compute the serie return

(def ts (timeserie @trade-price "refinitiv" "BNPP.PA" 2020-03-02T09:10 2020-03-02T17:30))
(/ ts (vget ts))

Tick Time Series dedicated functions

backward

parameter

clockref aka cf

parameter

delta

parameter

fields

parameter

forward

parameter

graphsample

parameters

example

hist

parameter

example

history

*parameter** code: a code related to source (e.g. a RIC like “BNPP.PA”)

keep

parameters

examples

Kurtosis

parameter

example

label

parameters

localize

parameters

examples

map

parameter

examples

map-reduce

parameter

example

map-reduce-arg

parameter

max

parameters

example

mean

parameter

example

median

parameter

example

merge

parameters

min

parameters

now

example

one

parameter

perimeter

parameter

prune

parameter

range

parameter

example

reverse

parameters

rts (recursive timeserie)

parameters

example

sd

parameter

example

sigma (+ with only one timeserie)

parameter

example

sign

parameter

skewness

parameter

example

slice

parameters