-the built in ssl cert generater was deprecated so it needed to be
updated or removed. I opted to calling openssl via external
command line execution. doing it this way make it easier to
update if deprecated again and reduces dependancy on libssl.
-tweaked the documentation a little.
-updated build.py to explicitly get the plugins directoy instead
of deriving it from the bin directory.
The host will no longer store logs in the database. Instead,
logging is done by syslog if running linux. If running
Windows, logs are now stored in a local file in the app
config directory.
Fixed a bug with -add_admin that would fail to create more
than one admin accounts in sequence due to the blank email
addresses being blank. It will now create fake email
addresses unique to each admin account.
Added -res_pw to reset user acccount passwords from the CLI
if needed.
Logging was also expanded to capture and log all failures
reported by all modules stderr output.
Updated build.py and install.py for QT6 support and moved
the linux .service file from /etc to /lib to conform to
systemd standards.
Removed the ls_dbg command because in database logging is
no longer done.
- I decided to remove the entire concept of a root user.
Instead, the host initializes as a blank slate and it
will be up to the host admin to create a rank 1 user via
the new command line option "-add_admin" to do initial
setup with.
- There is no longer such a concept as a protected user.
Meaning even the last rank 1 user in the host database
is allowed to delete or modify the rank of their own
account. To prevent permanent "admin lock out" in this
scenario the "-elevate" command line option was created.
- Host settings are no longer stored in the database.
Instead, host settings are now stored in a conf.json file
in /etc/mrci/conf.json if running on a linux based OS or
in %Programdata%\mrci\conf.json if running on Windows.
- Email templates are no longer stored in the database.
Instead, the templates can be any file formatted in UTF-8
text stored in the host file system. The files they point
to can be modified in the conf.json file.
- The conf file also replaced all use env variables so
MRCI_DB_PATH, MRCI_WORK_DIR, MRCI_PRIV_KEY and
MRCI_PUB_KEY are no longer in use. SSL/TLS cert paths can
be modified in the conf file.
- Removed email template cmds set_email_template and
preview_email.
- Also removed cmds close_host, host_config and
restart_host. The actions these commands could do is best
left to the host system command line.
- The database class will now explicitly check for write
permissions to the database and throw an appropriate
error message if the check fails. "DROP TABLE" SQL
abilities were added to make this happen.
- Removed async cmds exit(3), maxses(5) and restart(11).
fixed a bug in windows that caused the application to
fail detect that there is a host instance running
when there is indeed a host instance running.
also fixed a compile bug that cause QtCore to double
import and in turn call out duplicate definition
errors in windows when including applink.c.
changed the TEXT type id format from UTF16LE to UTF8 (no BOM). this
included all displayable text used throughout this project. doing
this reduced overhead because UTF16 strings required 2 bytes per
char while UTF8 runs on 1 byte for char. this string format also
expands support for QTs built in Postgresql driver or any driver
that supports UTF8 unicode only.
added "mod_instructions" to the client header format so the client
applications themselves can send direct command line args to the
modules loaded by the host. note: main.cpp needed to be modified
to check for core module parameters before checking the additional
parameters sent by the client; doing this protects against possible
unintentional core parameters being sent by the client.
added a Q_OS_WINDOWS check to applink.c so this file is completely
ignored when compiling on a Linux platform.
all commands that send emails will now return an appropriate error
message if the internal email client fails for any reason.
added a db_settings.json file that the host will now check for to
get database parameters such as hostname, username, driver and
password. doing this opens up the host to other database drivers
other than SQLITE. if not present, the host will create a default
db_settings file that uses SQLITE.
added the -ls_sql_drvs command line arg that will list all of the
SQL drivers that the host currently has installed and can be used
in the db_settings file.
- added a -host_trig command line option that creates a new host
instance without blocking user input.
- the build/install python scripts will no longer internally build
platform specific shell scripts; instead, they will now copy the
necessary templates from the /templates folder.
- modified both build.py and install.py to include windows
deployment support so this app can now be built/installed on
windows based host. note: i decided to have it "daemonized" as
a scheduled task that runs on system start instead of service
because it is simply easier to do it this way and will not
involve any 3rd party libraries or apps.
- added the src/applink.c file to the source code since it is
needed for windows openssl support.
- changed over all references of "endl" to "Qt::endl" to avoid
depreciation errors/warnings.
Fixed the shebang for the python build/install scripts to properly
point to the correct python executable in linux.
Fixed a bug that caused the host session to return false "command
not found errors" after running any command shortly after
successfully connecting.
- mutiple SSL cert files can now be added to the MRCI_PUB_KEY env variable
via colon seperated ':' path strings to complete the cert chain if such
a thing is required. It is no longer necessary to merge to multiple
certs into one to complete a cert chain.
- added -load_ssl command line option so cert data can be re-loaded in
real time without the need to stop-start the host.
- added more detailed error messages to the SSL loading process for
easier debugging.
- major changes to the build system include the use of python scripts
instead of the linux shell script file.
- linux_build.sh was removed since it is no longer needed.
- the new build process now run 2 python scripts: build.py and then
install.py.
- the resulting installer if built no longer uses makeself. the
installation and/or self extracting process is now handled entirely
by python and the install.py script.
The main reason for this change is to lay the ground work for multi-
platform support. It is still linux only for now but adding windows
support will be much easier in the future thanks to python's cross-
platform support.
- SSL certs are no longer stored in the host database. This was
done not only for security reasons but there is simply no need
to do such thing anymore.
- The host will longer support multiple SSL certs and will instead
have just a single cert for all TCP connections. This required a
change to the client header format that simply replaced the the
common name with padding. The host will also no longer send the
HOST_CERT type id during session initialization. HOST_CERT was
also removed as a type id.
- The cert and privite key are now pointed to files in the local
file system by the environment variables: MRCI_PRIV_KEY and
MRCI_PUB_KEY.
- The host will still create a default self-signed cert if a valid
cert and private key is not defined in the above environmental
vars. Since the host only support single certs now, the default
cert needed to be expanded to include subject alternative names.
The host will try to detect it's WAN ip address using ipify.org
and then assign SANs for all detected local LAN interfaces.
- Since the cert is now handled by environmental vars and nothing
related to it stored in the database, all the core commands
related to cert management were removed.
- fixed a bug that prevented the creator of a channel from being
added to it's member list as the owner.
- fixed a bug that caused the host to open all sub-channels in
read only mode.
- fixed a bug that caused mod processes to not get called to
shutdown when the session is closed.
- removed the internal module idle timer, this removed an
un-needed redundancy. with that, the idle timeout for mod
processes are now 5 secs while the timeout for cmd processes
remain at 2 mins.
The byte format positions and lens for the PEER_INFO and MY_INFO
type ids were fixed to properly reflect format(s).
Few format changes to the code using the 'auto' keyword. This will
be an ongoing thing on every commit until all parts of the code
use the 'auto' keyword were appropriate.
Changed the versioning system to a 4 number system that have the first 2
numbers as major.minor for the host application itself and the next 2
numbers (tcp_rev.mod_rev) used by clients and modules to determine
compatibility. A full description of this new system has been edited
into protocol.md. This new system offically starts at v3.0.0.0.
Added the PROMPT data type id that will work exactly like PRIV_TEXT except
it tells the client that the command is asking for non-private information
from the user.
Added PROG and PROG_LAST type ids that can be used by commands to notify
the client of the progress of the command if it is long running. The
long running fs_* commands were updated to use these instead of TEXT for
progress updates.
PUB_IPC, PRIV_IPC and PUB_IPC_WITH_FEEDBACK have all been combined into
one: ASYNC_PAYLOAD. This type id is now the only means at which module
commands can now run async commands. The command process object will
now determine where to direct the async payload (public, private or
public with feedback) based on the async command id being requested.
A description for TERM_CMD was missing in data_types.md so it was added.
Refactored HALT_CMD to YIELD_CMD. The new name just seems more appropriate
or the effect it has on the command.
Module commands can now do input hooking using the new ASYNC_HOOK_INPUT
and ASYNC_UNHOOK async commands. input hooking basically makes it so all
client data gets redirected to the module command that initiated the hook.
This can be used to implement something like a EULA agreement that blocks
all actions that can place during the session until the user accepts or
anything else to that effect.
The command process object will now check the open sub-channels list
being sent by ASYNC_CAST or ASYNC_LIMITED_CAST in any order and will not
be required match exactly to open sub-channels list in the session object.
It however cannot contain sub-channels not already listed in session's
list or else the async payload will be blocked.
Fixed the CmdProcess::validAsync() function that was comparing the input
aysnc command id with the process's command id in some places which is
invalid logic for this function.
Fixed the 'cast' core command that was outputting a malformed async
payload that didn't include the open writable sub-channels list.
Fixed a bug that caused all casted payloads to be forwared to the clients
even when the sub-channel(s) are closed.
Fixed the 'set_disp_name' core command so it can now see the -new_name
argument properly.
security updates:
various commands that change or create account passwords now disallow
the user name, display name or email from being contained in it. this
will force users to not use public information inside the password,
hardening password security a little.
the root user name is now changeable and required to be changed on
initial login. this harden security a little by giving host admins the
option to not have a well known user name attached to a high privileged
account.
users can no longer re-use the old password and/or user name when
required to change. however, this does not actually implement full
password history because the user can then later change the password
back to the old password after the required change.
the host can longer block by ip addresses and the auto block threshold
setting has been removed. something like this is best left up to
firewalls, routers, switches or any other networking infrastructure. in
the future i can consider adding event triggering that run certain
admin defined external or internal commands when the host detects
certain event thresholds.
minor changes/bug fixes:
all commands that change or create user names now no longer accept
user names that looks like an mail address. this works out better for
clients when differentiating logging in via user name or email address.
the recover_acct command now also have cancel on blank text options
making it more consistent with all other commands that take text input.
resetting the root user's account password via command line now also
unlocks it if locked.
the -help and -about command line options no longer display the
default password. a new -default_pw option was added for this purpose.
the -status -addr or -stop command line options require super user
privileges to run properly depending on how the host is installed.
an error message like "permission denied" was addded on failure to
make this requirement clear to the end user.
fs_copy and fs_move now does implicit skip on error instead of stop on
error.
the IDLE frame type id now carry an integer return code that can be
interpreted by clients to determine the result of the command that was
sent to the host.
house keeping:
all documentation was updated to reflect the changes made in this commit.
the module tester example is no longer relevant to this project so it
was deleted.
changed the default installattion used in the linux_build.sh
script to create an installer that deauly installs the
application data files in the /var fold folder instead of
the user specific folder in /home.
also made it so it will now create a system user named after
the name of the application and use that user to run the
systemd service instead of the currently logged in user.
doing this harded system security by having the application
run as an under privileged user. this also opens up the
possibility to lock down direct writing access to the host
database to just the internal module (in a future update).
removed the -start command line argument and have just the
-host option to start new host instances. the -start option
would start the host as the current user so i decided to
remove it as a way to encourage end users to start/stop the
host via systemd.
genfile commands would not terminate properly or
would end up de-synced with the host in one way or
another.
to fix this, i updated the GEN_FILE data type and
sub-protocol to now define the commands as download
or upload on the NEW_CMD frame so clients can now
define the direction of the GEN_FILE data of the
various GEN_FILE commands at the very start instead
of trying to determine that at run time.
also fixed this by creating the onTerminate() virtual
function in CmdObject and have it call this function
when term() is called. this makes it possible to
properly put the command object in a reset state if
using parameters outside of the base class when
term() is called.
updated all documentation related to the GEN_FILE
sub-protocol to reflect these changes.
fixed ASYNC_DEBUG_TEXT to self correct the ipc type
to PRIV_IPC so no debug messages can accidentally be
sent to peers.
fixed some errors found in the documentation and fully updated the internal
commands docs as well.
also updated the internal module to now respond to KILL_CMD so it can now be
signaled to terminate gracefully.
-the add_mod command now requires the -mod_path argument as it should.
-the session now filter out bad NEW_CMD frames based on bad command names and incorrect
frame sizes.
-modules running in list mode can now send ERR frames to the session to log error
messages to the host database.
Made some major changes to the project to facilitate a lighter code base and the
must flexible module interface possible.
-the mutli-process architecture now operate at the command object level so each
command now operate in it's own process instead of a single process handling
multiple command objects.
-each module is now an independent application that will now tell the session
object all of the commands it can run via named pipe. during command execution,
it will run the requested command object also running io with the session object
via named pipe.
with this change, it is now possible for modules to be developed in different
versions or QT or entirely different languages. the only requirement is the need
to support named pipes. shared memory segments is also a nice to have but not
needed.
-clients can now run multiple instances of the same command via changes to the
protocol. mrci frames will now include a branch id along with the command id.
the branch id can be used by clients to differentiate the io between instances
of the same command.
-the command states are longer controlled by a single object. it will now be up
to the command object (internal/exterenal) to send an IDLE frame to the client
to notify it that the command has finished. the session object will still track
if the command is in idle state or not but not directly control it.
-must async commands now use binary formatted data instead of TEXT as a way to
reduce overhead.
-all command objects can now send async commands. it is no longer limited to just
internal commands, however; the data of these async commands are verified by
session in some way to prevent host crashing due to malformed data.
-changed up the database structure to rely more on user ids, channel ids and
removed all foreign keys pointing to user names, channel names and sub-channel
names. also removed the groups table altogether. instead, the host rank is now
directly attached to the user data in the users table.
-changed the query object to now support the INNER JOIN SQL clause. this change
was needed to support the new database structure.
-version negotiation is now one-way via tcp connection or module interface.
the host will make it's own verion numner known to the client connected via
tcp or the module connected via named pipe. it will now be entirely up to the
client or module to decide if they support the host. another change in this
regard is the removal of the import rev for the modules. compatibility for
modules shall now use just the host verion.
-removed ls_cmds and cmd_info. the NEW_CMD frame now carries all information
about the command (cmd_id, cmd_name, summery, io and full_description) so it
is now possible for the clients to display the command documentation instead
of the host.
Documentation for the internal commands were updated to reflect the changes but
all other documentation will need to be updated in the near future.
Added a new fs_tree command that list all files and directories in a directory
tree. Just like fs_list, it has the option to output human readable text or
FILE_INFO frames. Also added the option to hide hidden files for both commands.
1.1.2 --> 1.1.3
Moved away from QPluginLoader to C/C++ style QLibrary files for the
module interface. It's less limiting this way because the host can
be built statically and still beable to load non-statically built
modules. It's not tested but the i think it should now beable to
load modules built on a different version of Qt than the host but
the module would must likely need to be build statically.
Another change to the module interface makes it so the module can
can now also see the host version in CommandLoader::hostRevOk()
and decide if the host can continue loading it.
The host 'host_info' internal command now displays the version of
Qt it was built on.
All documentation related to the module interface has been
updated. No attempt will be made at the host side to support rev
2 modules so the new minimum rev is now 3.
1.0.2 --> 1.1.2
$qt_dir in the build script will now default to the hard coded standard
directory for Qt binaries '/usr/lib/x86_64-linux-gnu/qt5/bin' if a valid
path is not provided by the user. End users that have customized setups
of Qt might need to modify this script to make sure $qt_dir points to the
correct directory at all times.
Moved the LICENSE.md file to the root directory so github can read it.
The ASYNC_USER_DELETED async command will now be forwarded to the client
if needed so clients can now be notified if the user account gets
deleted.
1.0.1 --> 1.0.2
Fixed a bug that caused the host to not respond to any commands when a
module is present. CmdExecutor::loadMods() had a malformed 'for' loop
that caused the command executor to infinite loop.
Also re-formed the way the external command loaders (modules) are
constructed. Instead, the modules are also intialized at the same time
as the internal command loader.
The CmdExecutor::loadModFile() and CmdExecutor::unloadModFile()
functions now load and unload modules via the module name instead of the
path to the module main file. The documentation for ASYNC_ENABLE_MOD and
ASYNC_DISABLE_MOD updated to reflect this change.
Also part of the module contruction re-form, it fixed a bug that caused
the command loaders to operate in a thread different from the command
executor. CmdExecutor::loadMods() was refactored into
cmdExecutor::buildCmdLoaders(), added the internal command loader to it
and have it so this function gets called only when the command executor
successfully moves to it's own thread.
Fixed a bug that caused the session to crash only when a module is
present. It turns out the qobject_cast() function used in
CmdExecutor::loadModFile() only pulled the surface class
ModCommandLoader so any attempt to access the base class functions from
CommandLoader caused the session to crash so to fix this,
ModCommandLoader was merged into CommandLoader. The Command loader
documentation and test module were updated to reflect this change.
Fixed a bug that auto removed any modules that were installed when the
host is restarted. The issue was in TCPServer::syncModPath() that used
the modules's main file without a suffix when the file did indeed have
a suffix, causing the function to think the module no longer exists and
remove all of it's files from the host file system. The original intent
for this function was to keep the module database table in sync with
what was actually in the host file system. It was decided that this is
not necessary so the function was removed altogether.
Also changed up the way the module files are deleted. The TCPSever class
will now use the module name instead of the path to the module's main
library file and will use a single use ModDeleteTimer for each module
that was requested to be deleted instead of a single continuous timer
constantly checking for modules to delete.
The ls_cmds command will now list the commands in alphabetical order of
the command names and fixed a bug with the -find option that would
display nothing if an exact match to any text in the table could not be
found.
Fixed a bug found in the deployed application that would segmentation
fault on startup due to certain library files being included in the
deployed installation directory. Updated the linux_build.sh script to
include only libsqlite, libQT5, libssl and libicu files. It will no
longer blindly copy all library files from the dev machine and then
remove files the setup script thinks the application might or might not
need.
Any module built on import rev1 will simply not work and no attempt will
be made to make it work so the host new minimum import rev is now rev2.
this update is mustly just bug fixes; nothing that would require any
client behaviour changes so just the host patch is bumped.
1.0.0 --> 1.0.1
