MRCI/docs/protocol.md

99 lines
5.4 KiB
Markdown
Raw Normal View History

2019-09-06 23:43:07 -04:00
### 1.1 The Protocol ###
Slimmed down and simplified host administering - I decided to remove the entire concept of a root user. Instead, the host initializes as a blank slate and it will be up to the host admin to create a rank 1 user via the new command line option "-add_admin" to do initial setup with. - There is no longer such a concept as a protected user. Meaning even the last rank 1 user in the host database is allowed to delete or modify the rank of their own account. To prevent permanent "admin lock out" in this scenario the "-elevate" command line option was created. - Host settings are no longer stored in the database. Instead, host settings are now stored in a conf.json file in /etc/mrci/conf.json if running on a linux based OS or in %Programdata%\mrci\conf.json if running on Windows. - Email templates are no longer stored in the database. Instead, the templates can be any file formatted in UTF-8 text stored in the host file system. The files they point to can be modified in the conf.json file. - The conf file also replaced all use env variables so MRCI_DB_PATH, MRCI_WORK_DIR, MRCI_PRIV_KEY and MRCI_PUB_KEY are no longer in use. SSL/TLS cert paths can be modified in the conf file. - Removed email template cmds set_email_template and preview_email. - Also removed cmds close_host, host_config and restart_host. The actions these commands could do is best left to the host system command line. - The database class will now explicitly check for write permissions to the database and throw an appropriate error message if the check fails. "DROP TABLE" SQL abilities were added to make this happen. - Removed async cmds exit(3), maxses(5) and restart(11).
2020-11-10 14:47:00 -05:00
The main goal of this application is to transport data from remote TCP clients to the [Modules](modules.md) defined in the host. How the data is processed and/or returned back to the client depends entirely on the module itself. The data format that the host understands for data transport is referred to as MRCI frames described below in section 1.2.
2019-09-06 23:43:07 -04:00
Before any MRCI frames can be transported, both the host and client need basic information about each other. This is done by having the client send a fixed length client header when it successfully connects to the host and the host will reply with it's own fixed length host header. Descriptions of these headers can be found in sections 1.4 and 1.5.
2019-09-06 23:43:07 -04:00
### 1.2 MRCI Frames ###
```
[type_id][cmd_id][branch_id][data_len][payload]
2019-09-06 23:43:07 -04:00
type_id - 1byte - 8bit little endian integer type id of the payload.
cmd_id - 2bytes - 16bit little endian integer command id.
branch_id - 2bytes - 16bit little endian integer branch id.
data_len - 3bytes - 24bit little endian integer size of the payload.
payload - variable - the actual data to be processed.
2019-09-06 23:43:07 -04:00
```
notes:
* A full description of the type id's can be found in the [Type_IDs.md](type_ids.md) document.
Slimmed down and simplified host administering - I decided to remove the entire concept of a root user. Instead, the host initializes as a blank slate and it will be up to the host admin to create a rank 1 user via the new command line option "-add_admin" to do initial setup with. - There is no longer such a concept as a protected user. Meaning even the last rank 1 user in the host database is allowed to delete or modify the rank of their own account. To prevent permanent "admin lock out" in this scenario the "-elevate" command line option was created. - Host settings are no longer stored in the database. Instead, host settings are now stored in a conf.json file in /etc/mrci/conf.json if running on a linux based OS or in %Programdata%\mrci\conf.json if running on Windows. - Email templates are no longer stored in the database. Instead, the templates can be any file formatted in UTF-8 text stored in the host file system. The files they point to can be modified in the conf.json file. - The conf file also replaced all use env variables so MRCI_DB_PATH, MRCI_WORK_DIR, MRCI_PRIV_KEY and MRCI_PUB_KEY are no longer in use. SSL/TLS cert paths can be modified in the conf file. - Removed email template cmds set_email_template and preview_email. - Also removed cmds close_host, host_config and restart_host. The actions these commands could do is best left to the host system command line. - The database class will now explicitly check for write permissions to the database and throw an appropriate error message if the check fails. "DROP TABLE" SQL abilities were added to make this happen. - Removed async cmds exit(3), maxses(5) and restart(11).
2020-11-10 14:47:00 -05:00
* Modules call commands via a command name but the host will assign a unique command id to all command names so clients can call them using a simple 2 byte integer instead of full text. The command ids can change as the modules change so it is recommended for clients to not depend on consistant command ids but depend on the [ASYNC_ADD_CMD](async.md) and [ASYNC_RM_CMD](async.md) async commands instead.
* The branch id is an id that can be assigned by the client itself to run muliple instances of the same command. Commands sent by a certain branch id will result in data sent back to the client from the module with that same branch id.
2019-09-06 23:43:07 -04:00
### 1.3 Versioning System ###
New versioning system, type ids and a few bug fixes Changed the versioning system to a 4 number system that have the first 2 numbers as major.minor for the host application itself and the next 2 numbers (tcp_rev.mod_rev) used by clients and modules to determine compatibility. A full description of this new system has been edited into protocol.md. This new system offically starts at v3.0.0.0. Added the PROMPT data type id that will work exactly like PRIV_TEXT except it tells the client that the command is asking for non-private information from the user. Added PROG and PROG_LAST type ids that can be used by commands to notify the client of the progress of the command if it is long running. The long running fs_* commands were updated to use these instead of TEXT for progress updates. PUB_IPC, PRIV_IPC and PUB_IPC_WITH_FEEDBACK have all been combined into one: ASYNC_PAYLOAD. This type id is now the only means at which module commands can now run async commands. The command process object will now determine where to direct the async payload (public, private or public with feedback) based on the async command id being requested. A description for TERM_CMD was missing in data_types.md so it was added. Refactored HALT_CMD to YIELD_CMD. The new name just seems more appropriate or the effect it has on the command. Module commands can now do input hooking using the new ASYNC_HOOK_INPUT and ASYNC_UNHOOK async commands. input hooking basically makes it so all client data gets redirected to the module command that initiated the hook. This can be used to implement something like a EULA agreement that blocks all actions that can place during the session until the user accepts or anything else to that effect. The command process object will now check the open sub-channels list being sent by ASYNC_CAST or ASYNC_LIMITED_CAST in any order and will not be required match exactly to open sub-channels list in the session object. It however cannot contain sub-channels not already listed in session's list or else the async payload will be blocked. Fixed the CmdProcess::validAsync() function that was comparing the input aysnc command id with the process's command id in some places which is invalid logic for this function. Fixed the 'cast' core command that was outputting a malformed async payload that didn't include the open writable sub-channels list. Fixed a bug that caused all casted payloads to be forwared to the clients even when the sub-channel(s) are closed. Fixed the 'set_disp_name' core command so it can now see the -new_name argument properly.
2020-03-08 14:58:51 -04:00
The host uses a 4 number versioning system that indicate rev numbers for the host application itself, the tcp interface and the module interface:
```
[Major][Minor][TCP_Rev][Mod_Rev]
3 . 2 . 1 . 0
New versioning system, type ids and a few bug fixes Changed the versioning system to a 4 number system that have the first 2 numbers as major.minor for the host application itself and the next 2 numbers (tcp_rev.mod_rev) used by clients and modules to determine compatibility. A full description of this new system has been edited into protocol.md. This new system offically starts at v3.0.0.0. Added the PROMPT data type id that will work exactly like PRIV_TEXT except it tells the client that the command is asking for non-private information from the user. Added PROG and PROG_LAST type ids that can be used by commands to notify the client of the progress of the command if it is long running. The long running fs_* commands were updated to use these instead of TEXT for progress updates. PUB_IPC, PRIV_IPC and PUB_IPC_WITH_FEEDBACK have all been combined into one: ASYNC_PAYLOAD. This type id is now the only means at which module commands can now run async commands. The command process object will now determine where to direct the async payload (public, private or public with feedback) based on the async command id being requested. A description for TERM_CMD was missing in data_types.md so it was added. Refactored HALT_CMD to YIELD_CMD. The new name just seems more appropriate or the effect it has on the command. Module commands can now do input hooking using the new ASYNC_HOOK_INPUT and ASYNC_UNHOOK async commands. input hooking basically makes it so all client data gets redirected to the module command that initiated the hook. This can be used to implement something like a EULA agreement that blocks all actions that can place during the session until the user accepts or anything else to that effect. The command process object will now check the open sub-channels list being sent by ASYNC_CAST or ASYNC_LIMITED_CAST in any order and will not be required match exactly to open sub-channels list in the session object. It however cannot contain sub-channels not already listed in session's list or else the async payload will be blocked. Fixed the CmdProcess::validAsync() function that was comparing the input aysnc command id with the process's command id in some places which is invalid logic for this function. Fixed the 'cast' core command that was outputting a malformed async payload that didn't include the open writable sub-channels list. Fixed a bug that caused all casted payloads to be forwared to the clients even when the sub-channel(s) are closed. Fixed the 'set_disp_name' core command so it can now see the -new_name argument properly.
2020-03-08 14:58:51 -04:00
Major - this indicate any changes to the host application that would cause
clients to need to change behaviour to maintain compatibility.
changes to the core command names, type id format changes, etc.
will cause the version major to increment.
Minor - this indicate any changes to the host application that clients will
not see and would not need behaviour changes to maintain
compatibility. documentation changes, bug fixes, security patches,
etc. will cause the version minor to increment.
TCP_Rev - this indicate any changes to the MRCI protocol that interface the
host with the clients via the TCP connection. any changes to the
MRCI frames, host/client headers, etc. will cause this rev to
increment.
Mod_Rev - this indicate any changes to the IPC protocol that interface the
host with the modules via named pipes. any changes to the IPC
Slimmed down and simplified host administering - I decided to remove the entire concept of a root user. Instead, the host initializes as a blank slate and it will be up to the host admin to create a rank 1 user via the new command line option "-add_admin" to do initial setup with. - There is no longer such a concept as a protected user. Meaning even the last rank 1 user in the host database is allowed to delete or modify the rank of their own account. To prevent permanent "admin lock out" in this scenario the "-elevate" command line option was created. - Host settings are no longer stored in the database. Instead, host settings are now stored in a conf.json file in /etc/mrci/conf.json if running on a linux based OS or in %Programdata%\mrci\conf.json if running on Windows. - Email templates are no longer stored in the database. Instead, the templates can be any file formatted in UTF-8 text stored in the host file system. The files they point to can be modified in the conf.json file. - The conf file also replaced all use env variables so MRCI_DB_PATH, MRCI_WORK_DIR, MRCI_PRIV_KEY and MRCI_PUB_KEY are no longer in use. SSL/TLS cert paths can be modified in the conf file. - Removed email template cmds set_email_template and preview_email. - Also removed cmds close_host, host_config and restart_host. The actions these commands could do is best left to the host system command line. - The database class will now explicitly check for write permissions to the database and throw an appropriate error message if the check fails. "DROP TABLE" SQL abilities were added to make this happen. - Removed async cmds exit(3), maxses(5) and restart(11).
2020-11-10 14:47:00 -05:00
frames or major changes to async commands will cause this rev to
New versioning system, type ids and a few bug fixes Changed the versioning system to a 4 number system that have the first 2 numbers as major.minor for the host application itself and the next 2 numbers (tcp_rev.mod_rev) used by clients and modules to determine compatibility. A full description of this new system has been edited into protocol.md. This new system offically starts at v3.0.0.0. Added the PROMPT data type id that will work exactly like PRIV_TEXT except it tells the client that the command is asking for non-private information from the user. Added PROG and PROG_LAST type ids that can be used by commands to notify the client of the progress of the command if it is long running. The long running fs_* commands were updated to use these instead of TEXT for progress updates. PUB_IPC, PRIV_IPC and PUB_IPC_WITH_FEEDBACK have all been combined into one: ASYNC_PAYLOAD. This type id is now the only means at which module commands can now run async commands. The command process object will now determine where to direct the async payload (public, private or public with feedback) based on the async command id being requested. A description for TERM_CMD was missing in data_types.md so it was added. Refactored HALT_CMD to YIELD_CMD. The new name just seems more appropriate or the effect it has on the command. Module commands can now do input hooking using the new ASYNC_HOOK_INPUT and ASYNC_UNHOOK async commands. input hooking basically makes it so all client data gets redirected to the module command that initiated the hook. This can be used to implement something like a EULA agreement that blocks all actions that can place during the session until the user accepts or anything else to that effect. The command process object will now check the open sub-channels list being sent by ASYNC_CAST or ASYNC_LIMITED_CAST in any order and will not be required match exactly to open sub-channels list in the session object. It however cannot contain sub-channels not already listed in session's list or else the async payload will be blocked. Fixed the CmdProcess::validAsync() function that was comparing the input aysnc command id with the process's command id in some places which is invalid logic for this function. Fixed the 'cast' core command that was outputting a malformed async payload that didn't include the open writable sub-channels list. Fixed a bug that caused all casted payloads to be forwared to the clients even when the sub-channel(s) are closed. Fixed the 'set_disp_name' core command so it can now see the -new_name argument properly.
2020-03-08 14:58:51 -04:00
increment.
```
2019-09-06 23:43:07 -04:00
New versioning system, type ids and a few bug fixes Changed the versioning system to a 4 number system that have the first 2 numbers as major.minor for the host application itself and the next 2 numbers (tcp_rev.mod_rev) used by clients and modules to determine compatibility. A full description of this new system has been edited into protocol.md. This new system offically starts at v3.0.0.0. Added the PROMPT data type id that will work exactly like PRIV_TEXT except it tells the client that the command is asking for non-private information from the user. Added PROG and PROG_LAST type ids that can be used by commands to notify the client of the progress of the command if it is long running. The long running fs_* commands were updated to use these instead of TEXT for progress updates. PUB_IPC, PRIV_IPC and PUB_IPC_WITH_FEEDBACK have all been combined into one: ASYNC_PAYLOAD. This type id is now the only means at which module commands can now run async commands. The command process object will now determine where to direct the async payload (public, private or public with feedback) based on the async command id being requested. A description for TERM_CMD was missing in data_types.md so it was added. Refactored HALT_CMD to YIELD_CMD. The new name just seems more appropriate or the effect it has on the command. Module commands can now do input hooking using the new ASYNC_HOOK_INPUT and ASYNC_UNHOOK async commands. input hooking basically makes it so all client data gets redirected to the module command that initiated the hook. This can be used to implement something like a EULA agreement that blocks all actions that can place during the session until the user accepts or anything else to that effect. The command process object will now check the open sub-channels list being sent by ASYNC_CAST or ASYNC_LIMITED_CAST in any order and will not be required match exactly to open sub-channels list in the session object. It however cannot contain sub-channels not already listed in session's list or else the async payload will be blocked. Fixed the CmdProcess::validAsync() function that was comparing the input aysnc command id with the process's command id in some places which is invalid logic for this function. Fixed the 'cast' core command that was outputting a malformed async payload that didn't include the open writable sub-channels list. Fixed a bug that caused all casted payloads to be forwared to the clients even when the sub-channel(s) are closed. Fixed the 'set_disp_name' core command so it can now see the -new_name argument properly.
2020-03-08 14:58:51 -04:00
Any increments to the Major resets the Minor to 0. Any 3rd party client applications connecting to a MRCI host need to be aware of this versioning but does not need to adopt it as it's own version number.
2019-09-06 23:43:07 -04:00
### 1.4 Client Header ###
```
2020-08-09 12:21:33 -04:00
[tag][appName][mod_instructions][padding]
2019-09-06 23:43:07 -04:00
tag - 4bytes - 0x4D, 0x52, 0x43, 0x49 (MRCI)
2020-08-09 12:21:33 -04:00
appName - 32bytes - UTF8 string (padded with 0x00)
modInst - 128bytes - UTF8 string (padded with 0x00)
padding - 128bytes - string of (0x00)
2019-09-06 23:43:07 -04:00
```
notes:
2020-08-09 12:21:33 -04:00
* **tag** is just a fixed ascii string "MRCI" that indicates to the host that the client is indeed attempting to use the MRCI protocol.
2019-09-06 23:43:07 -04:00
2020-08-09 12:21:33 -04:00
* **appName** is the name of the client application that is connected to the host. It can also contain the client's app version if needed because it doesn't follow any particular standard. This string is accessable to all modules so the commands themselves can be made aware of what app the user is currently using.
* **modInst** is an additional set of command lines that can be passed onto to all module processes when they are intialized. This can be used by certain clients that want to intruct certain modules that might be installed in the host to do certain actions during intialization. This remains constant for as long as the session is active and cannot be changed at any point.
2019-09-06 23:43:07 -04:00
### 1.5 Host Header ###
```
Format:
New versioning system, type ids and a few bug fixes Changed the versioning system to a 4 number system that have the first 2 numbers as major.minor for the host application itself and the next 2 numbers (tcp_rev.mod_rev) used by clients and modules to determine compatibility. A full description of this new system has been edited into protocol.md. This new system offically starts at v3.0.0.0. Added the PROMPT data type id that will work exactly like PRIV_TEXT except it tells the client that the command is asking for non-private information from the user. Added PROG and PROG_LAST type ids that can be used by commands to notify the client of the progress of the command if it is long running. The long running fs_* commands were updated to use these instead of TEXT for progress updates. PUB_IPC, PRIV_IPC and PUB_IPC_WITH_FEEDBACK have all been combined into one: ASYNC_PAYLOAD. This type id is now the only means at which module commands can now run async commands. The command process object will now determine where to direct the async payload (public, private or public with feedback) based on the async command id being requested. A description for TERM_CMD was missing in data_types.md so it was added. Refactored HALT_CMD to YIELD_CMD. The new name just seems more appropriate or the effect it has on the command. Module commands can now do input hooking using the new ASYNC_HOOK_INPUT and ASYNC_UNHOOK async commands. input hooking basically makes it so all client data gets redirected to the module command that initiated the hook. This can be used to implement something like a EULA agreement that blocks all actions that can place during the session until the user accepts or anything else to that effect. The command process object will now check the open sub-channels list being sent by ASYNC_CAST or ASYNC_LIMITED_CAST in any order and will not be required match exactly to open sub-channels list in the session object. It however cannot contain sub-channels not already listed in session's list or else the async payload will be blocked. Fixed the CmdProcess::validAsync() function that was comparing the input aysnc command id with the process's command id in some places which is invalid logic for this function. Fixed the 'cast' core command that was outputting a malformed async payload that didn't include the open writable sub-channels list. Fixed a bug that caused all casted payloads to be forwared to the clients even when the sub-channel(s) are closed. Fixed the 'set_disp_name' core command so it can now see the -new_name argument properly.
2020-03-08 14:58:51 -04:00
[reply][major][minor][tcp_rev][mod_rev][sesId]
2019-09-06 23:43:07 -04:00
New versioning system, type ids and a few bug fixes Changed the versioning system to a 4 number system that have the first 2 numbers as major.minor for the host application itself and the next 2 numbers (tcp_rev.mod_rev) used by clients and modules to determine compatibility. A full description of this new system has been edited into protocol.md. This new system offically starts at v3.0.0.0. Added the PROMPT data type id that will work exactly like PRIV_TEXT except it tells the client that the command is asking for non-private information from the user. Added PROG and PROG_LAST type ids that can be used by commands to notify the client of the progress of the command if it is long running. The long running fs_* commands were updated to use these instead of TEXT for progress updates. PUB_IPC, PRIV_IPC and PUB_IPC_WITH_FEEDBACK have all been combined into one: ASYNC_PAYLOAD. This type id is now the only means at which module commands can now run async commands. The command process object will now determine where to direct the async payload (public, private or public with feedback) based on the async command id being requested. A description for TERM_CMD was missing in data_types.md so it was added. Refactored HALT_CMD to YIELD_CMD. The new name just seems more appropriate or the effect it has on the command. Module commands can now do input hooking using the new ASYNC_HOOK_INPUT and ASYNC_UNHOOK async commands. input hooking basically makes it so all client data gets redirected to the module command that initiated the hook. This can be used to implement something like a EULA agreement that blocks all actions that can place during the session until the user accepts or anything else to that effect. The command process object will now check the open sub-channels list being sent by ASYNC_CAST or ASYNC_LIMITED_CAST in any order and will not be required match exactly to open sub-channels list in the session object. It however cannot contain sub-channels not already listed in session's list or else the async payload will be blocked. Fixed the CmdProcess::validAsync() function that was comparing the input aysnc command id with the process's command id in some places which is invalid logic for this function. Fixed the 'cast' core command that was outputting a malformed async payload that didn't include the open writable sub-channels list. Fixed a bug that caused all casted payloads to be forwared to the clients even when the sub-channel(s) are closed. Fixed the 'set_disp_name' core command so it can now see the -new_name argument properly.
2020-03-08 14:58:51 -04:00
reply - 1byte - 8bit little endian unsigned int
major - 2bytes - 16bit little endian unsigned int
minor - 2bytes - 16bit little endian unsigned int
tcp_rev - 2bytes - 16bit little endian unsigned int
mod_rev - 2bytes - 16bit little endian unsigned int
sesId - 28bytes - 224bit sha3 hash
2019-09-06 23:43:07 -04:00
```
notes:
* **reply** is a numeric value that the host returns in it's header to communicate to the client if SSL need to initated or not.
2019-09-06 23:43:07 -04:00
* reply = 1, means SSL is not required so the client doesn't need to take any further action.
* reply = 2, means SSL is required to continue so the client needs to send a STARTLS signal.
2019-09-06 23:43:07 -04:00
* **sesId** is the session id. It is a unique 224bit sha3 hash generated against the current date and time of session creation (down to the msec) and the machine id of the host. This can be used by the host or client to uniquely identify the current session or past sessions.