Datastore core

Core Module - Datastore - A module used to store data in the global table with the option to have it sync to an external source.

Usage

-- Types of Datastore
-- This datastore will not save data externally and can be used to watch for updates on values within it
-- A common use might be to store data for a gui and only update the gui when a value changes
local LocalDatastore = Datastore.connect('LocalDatastore')

-- This datastore will allow you to use the save and request method, this allows you to have persistent data
-- Should be used over auto save as it creates less save requests, but this means you need to tell the data to be saved
-- We use this type for player data as we know the data only needs to be saved when the player leaves
local PersistentDatastore = Datastore.connect('PersistentDatastore', true) -- save_to_disk

-- This datastore is the same as above but the save method will be called automatically when ever you change a value
-- An auto save datastore should be used if the data does not change often, this can be global settings and things of that sort
-- If it is at all possible to setup events to unload and/or save the data then this is preferable
local AutosaveDatastore = Datastore.connect('AutosaveDatastore', true, true) -- save_to_disk, auto_save

-- Finally you can have a datastore that propagates its changes to all other connected servers, this means request does not need to be used
-- This should be used when you might have data conflicts while saving, this is done by pushing the saved value to all active servers
-- The request method has little use after server start as any external changes to the value will be pushed automatically
-- Auto save can also be used with this type and you should follow the same guidelines above for when this should be avoided
local PropagateDatastore = Datastore.connect('PropagateDatastore', true, false, true) -- save_to_disk, propagate_changes

-- Using Datastores Locally
-- Once you have your datastore connection setup, any further requests with connect will return the same datastore
-- This is important to know because the settings passed as parameters you have an effect when it is first created

-- One useful thing that you might want to set up before runtime is a serializer, this will convert non string keys into strings
-- This serializer will allow use to pass a player object and still have it serialized to the players name
local ExampleData = Datastore.connect('ExampleData')
ExampleData:set_serializer(function(rawKey)
    return rawKey.name
end)

-- If we want to get data from the datastore we can use get or get_all
local value = ExampleData:get(player, defaultValue)
local values = ExampleData:get_all()

-- If we want to set data then we can use set, increment, update, or update_all
ExampleData:set(player, 10)
ExampleData:increment(player)
ExampleData:update(player, function(player_name, value)
    return value * 2
end)
ExampleData:update_all(function(player_name, value)
    return value * 2
end)

-- If we want to remove data then we use remove
ExampleData:remove(player)

-- We can also listen for updates to a value done by any of the above methods with on_update
ExampleData:on_update(function(player_name, value)
    game.print(player_name..' has had their example data updated to '..tostring(value))
end)

-- Using Datastore Externally
-- If save_to_disk is used then this opens up the option for persistent data which you can request, save, and remove
-- All of the local methods are still usable put now there is the option for extra events
-- In order for this to work there must be an external script to read datastore.pipe and inject with Datastore.ingest

-- To request data you would use request and the on_load event, this event can be used to modify data before it is used
ExampleData:request(player)
ExampleData:on_load(function(player_name, value)
    game.print('Loaded example data for '..player_name)
    -- A value can be returned here to overwrite the received value
end)

-- To save data you would use save and the on_save event, this event can be used to modify data before it is saved
ExampleData:save(player)
ExampleData:on_save(function(player_name, value)
    game.print('Saved example data for '..player_name)
    -- A value can be returned here to overwrite the value which is saved
end)

-- To remove data locally but not externally, like if a player logs off, you would use unload and on_unload
ExampleData:unload(player)
ExampleData:on_unload(function(player_name, value)
    game.print('Unloaded example data for '..player_name)
    -- Any return is ignored, this is event is for cleaning up other data
end)

-- Using Datastore Messaging
-- The message action can be used regardless of save_to_disk being set as no data is saved, but an external script is still required
-- These messages can be used to send data to other servers which doesnt need to be saved such as shouts or commands
-- Using messages is quite simple only using message and on_message
ExampleData:message(key, message)
ExampleData:on_message(function(key, message)
    game.print('Received message '..message)
end)

-- Combined Datastores
-- A combined datastore is a datastore which stores its data inside of another datastore
-- This means that the data is stored more efficiently in the external database and less requests need to be made
-- To understand how combined datastores work think of each key in the parent as a table where the sub datastore is a key in that table
-- Player data is the most used version of the combined datastore, below is how the player data module is setup
local PlayerData = Datastore.connect('PlayerData', true) -- saveToDisk
PlayerData:set_serializer(Datastore.name_serializer) -- use player name as key
PlayerData:combine('Statistics')
PlayerData:combine('Settings')
PlayerData:combine('Required')

-- You can then further combine datastores to any depth, below we add some possible settings and statistics that we might use
-- Although we dont in this example, each of these functions returns the datastore object which you should use as a local value
PlayerData.Settings:combine('Color')
PlayerData.Settings:combine('Quickbar')
PlayerData.Settings:combine('JoinMessage')
PlayerData.Statistics:combine('Playtime')
PlayerData.Statistics:combine('JoinCount')

-- Because sub datastore work just like a normal datastore you dont need any special code, using get and set will still return as if it wasnt a sub datastore
-- Things like the serializer and the datastore settings are always the same as the parent so you dont need to worry about setting up the serializer each time
-- And because save, request, and unload methods all point to the root datastore you are able to request and save your data as normal

-- If you used get_all on PlayerData this is what you would get:
{
    Cooldude2606 = {
        Settings = {
            Color = 'ColorValue',
            Quickbar = 'QuickbarValue',
            JoinMessage = 'JoinMessageValue'
        },
        Statistics = {
            Playtime = 'PlaytimeValue',
            JoinCount = 'JoinCountValue'
        }
    }
}

-- If you used get_all on PlayerData.Settings this is what you would get:
{
    Cooldude2606 = {
        Color = 'ColorValue',
        Quickbar = 'QuickbarValue',
        JoinMessage = 'JoinMessageValue'
    }
}

-- If you used get_all on PlayerData.Settings.Color this is what you would get:
{
    Cooldude2606 = 'ColorValue'
}

Dependencies

utils.event

Fields

global.datastores	Save datastores in the global table

Datastore Manager

metatable	Metatable used on datastores
connect(datastoreName[, saveToDisk=false][, autoSave=false][, propagateChanges=false])	Make a new datastore connection, if a connection already exists then it is returned
combine(datastoreName, subDatastoreName)	Make a new datastore that stores its data inside of another one
ingest(action, datastoreName, key, valueJson)	Ingest the result from a request, this is used through a rcon interface to sync data
debug([datastoreName])	Debug, Use to get all datastores, or return debug info on a datastore
name_serializer(rawKey)	Commonly used serializer, returns the name of the object

Datastore Internal

debug()	Debug, Get the debug info for this datastore
raw_get(key[, fromChild=false])	Internal, Get data following combine logic
raw_set(key, value)	Internal, Set data following combine logic
serialize(rawKey)	Internal, Return the serialized key
write_action(action, key, value)	Internal, Writes an event to the output file to be saved and/or propagated

Datastore Local

combine(subDatastoreName)	Create a new datastore which is stores its data inside of this datastore
set_serializer(callback)	Set a callback that will be used to serialize keys which aren't strings
set_default(value, allowSet)	Set a default value to be returned by get if no other default is given, using will mean get will never return nil, set using the default will set to nil to save space
set_metadata(tags)	Set metadata tags on this datastore which can be accessed by other scripts
get(key[, default])	Get a value from local storage, option to have a default value, do not edit the data returned as changes may not save, use update if you want to make changes
set(key, value)	Set a value in local storage, will trigger on_update then on_save, save_to_disk and auto_save is required for on_save
increment(key[, delta=1])	Increment the value in local storage, only works for number values, will trigger on_update then on_save, save_to_disk and auto_save is required for on_save
update(key, callback)	Use a function to update the value locally, will trigger on_update then on_save, save_to_disk and auto_save is required for on_save
remove(key)	Remove a value locally and on the external source, works regardless of propagateChanges, requires save_to_disk for external changes
get_all([callback])	Get all keys in this datastore, optional filter callback
update_all(callback)	Update all keys in this datastore using the same update function

Datastore External

request(key)	Request a value from an external source, will trigger on_load when data is received
save(key)	Save a value to an external source, will trigger on_save before data is saved, save_to_disk must be set to true
unload(key)	Save a value to an external source and remove locally, will trigger on_unload then on_save, save_to_disk is not required for on_unload
message(key, message)	Use to send a message over the connection, works regardless of saveToDisk and propagateChanges
save_all([callback])	Save all the keys in the datastore, optional filter callback
unload_all([callback])	Unload all the keys in the datastore, optional filter callback

Events

raise_event(event_name, key[, value][, old_value][, source])	Internal, Raise an event on this datastore
on_load	Register a callback that triggers when data is loaded from an external source, returned value is saved locally
on_save	Register a callback that triggers before data is saved, returned value is saved externally
on_unload	Register a callback that triggers before data is unloaded, returned value is ignored
on_message	Register a callback that triggers when a message is received, returned value is ignored
on_update	Register a callback that triggers any time a value is changed, returned value is ignored

Dependencies

# utils.event

Fields

# global.datastores

Save datastores in the global table

Datastore Manager

# metatable

Metatable used on datastores

Fields:

• __index
• __newidnex
• __call

# connect(datastoreName[, saveToDisk=false][, autoSave=false][, propagateChanges=false])

Make a new datastore connection, if a connection already exists then it is returned

Parameters:

• datastoreName : (string) The name that you want the new datastore to have, this can not have any whitespace
• saveToDisk : (boolean) When set to true, using the save method with write the data to datastore.pipe (default: false)
• autoSave : (boolean) When set to true, using any method which modifies data will cause the data to be saved (default: false)
• propagateChanges : (boolean) When set to true, using the save method will send the data to all other connected servers (default: false)

Returns:

• (table) The new datastore connection that can be used to access and modify data in the datastore

Usage:

-- Connecting to the test datastore which will allow saving to disk
local ExampleData = Datastore.connect('ExampleData', true) -- saveToDisk

# combine(datastoreName, subDatastoreName)

Make a new datastore that stores its data inside of another one

Parameters:

• datastoreName : (string) The name of the datastore that will contain the data for the new datastore
• subDatastoreName : (string) The name of the new datastore, this name will also be used as the key inside the parent datastore

Returns:

• (table) The new datastore connection that can be used to access and modify data in the datastore

Usage:

-- Setting up a datastore which stores its data inside of another datastore
local BarData = Datastore.combine('ExampleData', 'Bar')

# ingest(action, datastoreName, key, valueJson)

Ingest the result from a request, this is used through a rcon interface to sync data

Parameters:

• action : (string) The action that should be done, can be: remove, message, propagate, or request
• datastoreName : (string) The name of the datastore that should have the action done to it
• key : (string) The key of that datastore that is having the action done to it
• valueJson : (string) The json string for the value being ingested, remove does not require a value

Usage:

-- Replying to a data request
Datastore.ingest('request', 'ExampleData', 'TestKey', 'Foo')

# debug([datastoreName])

Debug, Use to get all datastores, or return debug info on a datastore

Parameters:

• datastoreName : (string) The name of the datastore to get the debug info of (optional)

Usage:

-- Get all the datastores
local datastores = Datastore.debug()

-- Getting the debug info for a datastore
local debug_info = Datastore.debug('ExampleData')

# name_serializer(rawKey)

Commonly used serializer, returns the name of the object

Parameters:

• rawKey : (any) The raw key that will be serialized, this can be things like player, force, surface, etc

Returns:

• (string) The name of the object that was passed

Usage:

-- Using the name serializer for your datastore
local ExampleData = Datastore.connect('ExampleData')
ExampleData:set_serializer(Datastore.name_serializer)

Datastore Internal

# debug()

Debug, Get the debug info for this datastore

Returns:

• (table) The debug info for this datastore, contains stuff like parent, settings, children, etc

Usage:

-- Get the debug info for a datastore
local ExampleData = Datastore.connect('ExampleData')
local debug_info = ExampleData:debug()

# raw_get(key[, fromChild=false])

Internal, Get data following combine logic

Parameters:

• key : (string) The key to get the value of from this datastore
• fromChild : (boolean) If the get request came from a child of this datastore (default: false)

Returns:

• (any) The value that was stored at this key in this datastore

Usage:

-- Internal, Get the data from a datastore
local value = self:raw_get('TestKey')

# raw_set(key, value)

Internal, Set data following combine logic

Parameters:

• key : (string) The key to set the value of in this datastore
• value : (any) The value that will be set at this key

Usage:

-- Internal, Set the value in a datastore
self:raw_set('TestKey', 'Foo')

# serialize(rawKey)

Internal, Return the serialized key

Parameters:

• rawKey : (any) The key that needs to be serialized, if it is already a string then it is returned

Returns:

• (string) The key after it has been serialized

Usage:

-- Internal, Ensure that the key is a string
key = self:serialize(key)

# write_action(action, key, value)

Internal, Writes an event to the output file to be saved and/or propagated

Parameters:

• action : (string) The action that should be wrote to datastore.pipe, can be request, remove, message, save, propagate
• key : (string) The key that the action is being preformed on
• value : (any) The value that should be used with the action

Usage:

-- Write a data request to datastore.pipe
self:write_action('request', 'TestKey')

-- Write a data save to datastore.pipe
self:write_action('save', 'TestKey', 'Foo')

Datastore Local

# combine(subDatastoreName)

Create a new datastore which is stores its data inside of this datastore

Parameters:

• subDatastoreName : (string) The name of the datastore that will have its data stored in this datastore

Returns:

• (table) The new datastore that was created inside of this datastore

Usage:

-- Add a new sub datastore
local ExampleData = Datastore.connect('ExampleData')
local BarData = ExampleData:combine('Bar')

# set_serializer(callback)

Set a callback that will be used to serialize keys which aren't strings

Parameters:

• callback : (function) The function that will be used to serialize non string keys passed as an argument

Usage:

-- Set a custom serializer, this would be the same as Datastore.name_serializer
local ExampleData = Datastore.connect('ExampleData')
ExampleData:set_serializer(function(rawKey)
    return rawKey.name
end)

# set_default(value, allowSet)

Set a default value to be returned by get if no other default is given, using will mean get will never return nil, set using the default will set to nil to save space

Parameters:

• value : (any) The value that will be deep copied by get if the value is nil and no other default is given
• allowSet : (boolean) When true if the default is passed as the value for set it will be set rather than setting nil

Usage:

-- Set a default value to be returned by get
local ExampleData = Datastore.connect('ExampleData')
ExampleData:set_default('Foo')

# set_metadata(tags)

Set metadata tags on this datastore which can be accessed by other scripts

Parameters:

• tags : (table) A table of tags that you want to set in the metadata for this datastore

Usage:

-- Adding metadata that could be used by a gui to help understand the stored data
local ExampleData = Datastore.connect('ExampleData')
ExampleData:set_metadata{
    caption = 'Test Data',
    tooltip = 'Data used for testing datastores',
    type = 'table'
}

# get(key[, default])

Get a value from local storage, option to have a default value, do not edit the data returned as changes may not save, use update if you want to make changes

Parameters:

• key : (any) The key that you want to get the value of, must be a string unless a serializer is set
• default : (any) The default value that will be returned if no value is found in the datastore (optional)

Usage:

-- Get a key from the datastore, the default will be deep copied if no value exists in the datastore
local ExampleData = Datastore.connect('ExampleData')
local value = ExampleData:get('TestKey')

# set(key, value)

Set a value in local storage, will trigger on_update then on_save, save_to_disk and auto_save is required for on_save

Parameters:

• key : (any) The key that you want to set the value of, must be a string unless a serializer is set
• value : (any) The value that you want to set for this key

Usage:

-- Set a value in the datastore, this will trigger on_update, if auto_save is true then will trigger save
local ExampleData = Datastore.connect('ExampleData')
ExampleData:set('TestKey', 'Foo')

# increment(key[, delta=1])

Increment the value in local storage, only works for number values, will trigger on_update then on_save, save_to_disk and auto_save is required for on_save

Parameters:

• key : (any) The key that you want to increment the value of, must be a string unless a serializer is set
• delta : (number) The amount that you want to increment the value by, can be negative or a decimal (default: 1)

Usage:

-- Increment a value in a datastore, the value must be a number or nil, if nil 0 is used as the start value
local ExampleData = Datastore.connect('ExampleData')
ExampleData:increment('TestNumber')

# update(key, callback)

Use a function to update the value locally, will trigger on_update then on_save, save_to_disk and auto_save is required for on_save

Parameters:

• key : (any) The key that you want to apply the update to, must be a string unless a serializer is set
• callback : (function) The function that will be used to update the value at this key

Usage:

-- Using a function to update a value, if a value is returned then this will be the new value
local ExampleData = Datastore.connect('ExampleData')
ExampleData:increment('TestKey', function(key, value)
    return value..value
end)

# remove(key)

Remove a value locally and on the external source, works regardless of propagateChanges, requires save_to_disk for external changes

Parameters:

• key : (any) The key that you want to remove locally and externally, must be a string unless a serializer is set

Usage:

-- Remove a key locally and externally
local ExampleData = Datastore.connect('ExampleData')
ExampleData:remove('TestKey')

# get_all([callback])

Get all keys in this datastore, optional filter callback

Parameters:

• callback : (function) The filter function that can be used to filter the results returned (optional)

Returns:

• (table) All the data that is in this datastore, filtered if a filter was provided

Usage:

-- Get all the data in this datastore
local ExampleData = Datastore.connect('ExampleData')
local data = ExampleData:get_all()

-- Get all the data in this datastore, with a filter
local ExampleData = Datastore.connect('ExampleData')
local data = ExampleData:get_all(function(key, value)
    return type(value) == 'string'
end)

# update_all(callback)

Update all keys in this datastore using the same update function

Parameters:

• callback : (function) The update function that will be applied to each key

Usage:

-- Get all the data in this datastore, with a filter
local ExampleData = Datastore.connect('ExampleData')
ExampleData:update_all(function(key, value)
    return value..value
end)

Datastore External

# request(key)

Request a value from an external source, will trigger on_load when data is received

Parameters:

• key : (any) The key that you want to request from an external source, must be a string unless a serializer is set

Usage:

-- Request a key from an external source, on_load is triggered when data is received
local ExampleData = Datastore.connect('ExampleData')
ExampleData:request('TestKey')

# save(key)

Save a value to an external source, will trigger on_save before data is saved, save_to_disk must be set to true

Parameters:

• key : (any) The key that you want to save to an external source, must be a string unless a serializer is set

Usage:

-- Save a key to an external source, save_to_disk must be set to true for there to be any effect
local ExampleData = Datastore.connect('ExampleData')
ExampleData:save('TestKey')

# unload(key)

Save a value to an external source and remove locally, will trigger on_unload then on_save, save_to_disk is not required for on_unload

Parameters:

• key : (any) The key that you want to unload from the datastore, must be a string unless a serializer is set

Usage:

-- Unload a key from the datastore, get will now return nil and value will be saved externally if save_to_disk is set to true
local ExampleData = Datastore.connect('ExampleData')
ExampleData:unload('TestKey')

# message(key, message)

Use to send a message over the connection, works regardless of saveToDisk and propagateChanges

Parameters:

• key : (any) The key that you want to send a message over, must be a string unless a serializer is set
• message : (any) The message that you want to send to other connected servers, or external source

Usage:

-- Send a message to other servers on this key, can listen for messages with on_message
local ExampleData = Datastore.connect('ExampleData')
ExampleData:message('TestKey', 'Foo')

# save_all([callback])

Save all the keys in the datastore, optional filter callback

Parameters:

• callback : (function) The filter function that can be used to filter the keys saved (optional)

Usage:

-- Save all the data in this datastore
local ExampleData = Datastore.connect('ExampleData')
local data = ExampleData:save_all()

-- Save all the data in this datastore, with a filter
local ExampleData = Datastore.connect('ExampleData')
ExampleData:save_all(function(key, value)
    return type(value) == 'string'
end)

# unload_all([callback])

Unload all the keys in the datastore, optional filter callback

Parameters:

• callback : (function) The filter function that can be used to filter the keys unloaded (optional)

Usage:

-- Unload all the data in this datastore
local ExampleData = Datastore.connect('ExampleData')
ExampleData:unload_all()

-- Unload all the data in this datastore, with a filter
local ExampleData = Datastore.connect('ExampleData')
ExampleData:unload_all(function(key, value)
    return type(value) == 'string'
end)

Events

# raise_event(event_name, key[, value][, old_value][, source])

Internal, Raise an event on this datastore

Parameters:

• event_name : (string) The name of the event to raise for this datastore
• key : (string) The key that this event is being raised for
• value : (any) The current value that this key has, might be a deep copy of the value (optional)
• old_value : (any) The previous value that this key has, might be a deep copy of the value (optional)
• source : (string) Where this call came from, used to do event recursion so can be parent or child (optional)

Returns:

• (any) The value that is left after being passed through all the event handlers

Usage:

-- Internal, Getting the value that should be saved
value = self:raise_event('on_save', key, value)

# on_load

Register a callback that triggers when data is loaded from an external source, returned value is saved locally

• callback : (function) The handler that will be registered to the on_load event

Usage:

-- Adding a handler to on_load, returned value will be saved locally, can be used to deserialize the value beyond a normal json
local ExampleData = Datastore.connect('ExampleData')
ExampleData:on_load(function(key, value)
    game.print('Test data loaded for: '..key)
end)

# on_save

Register a callback that triggers before data is saved, returned value is saved externally

• callback : (function) The handler that will be registered to the on_load event

Usage:

-- Adding a handler to on_save, returned value will be saved externally, can be used to serialize the value beyond a normal json
local ExampleData = Datastore.connect('ExampleData')
ExampleData:on_save(function(key, value)
    game.print('Test data saved for: '..key)
end)

# on_unload

Register a callback that triggers before data is unloaded, returned value is ignored

• callback : (function) The handler that will be registered to the on_load event

Usage:

-- Adding a handler to on_unload, returned value is ignored, can be used to clean up guis or local values related to this data
local ExampleData = Datastore.connect('ExampleData')
ExampleData:on_load(function(key, value)
    game.print('Test data unloaded for: '..key)
end)

# on_message

Register a callback that triggers when a message is received, returned value is ignored

• callback : (function) The handler that will be registered to the on_load event

Usage:

-- Adding a handler to on_message, returned value is ignored, can be used to receive messages from other connected servers without saving data
local ExampleData = Datastore.connect('ExampleData')
ExampleData:on_message(function(key, value)
    game.print('Test data message for: '..key)
end)

# on_update

Register a callback that triggers any time a value is changed, returned value is ignored

• callback : (function) The handler that will be registered to the on_load event

Usage:

-- Adding a handler to on_update, returned value is ignored, can be used to update guis or send messages when data is changed
local ExampleData = Datastore.connect('ExampleData')
ExampleData:on_update(function(key, value)
    game.print('Test data updated for: '..key)
end)