.
ctys-vhost - core address resolution interface and database front-end
ctys-vhost
[-c <spent cost on execution environment> =:( MINCNT|MAXCNT|CNT ) ] [-C <DB sources> =:( OFF |CLEARTMP |CLEARALL |GROUPS |KEEPALL |LIST |LISTCACHE |LISTARGETS |LISTGROUPS |MEMBERSDB |MACMAPONLY |MACMAP |REBUILDCACHE ) ] [-d <debug-level>] [-h] [-H <help-options>] [-i <input-list>=[CTYSADDRESS|CTYS]] [-I <0-9>] [-l <USER>] [-M <result-set-output-reduction> =:( FIRST|LAST|ALL|COMPLEMENT|SORT|USORT|UNIQUE ) ] [-o <output-list> =:( ( ( [ARCH][,] [ACCELERATOR|ACCEL][,] [CATEGORY|CAT][,] [CONTEXTSTRING|CSTRG][,] [CPORT|VNCPORT][,] [CTYSADDRESS|CTYS][,] [CTYSRELEASE][,] [DIST][,] [DISTREL][,] [EXECLOCATION][,] [EXEPATH|EXEP][,] [GATEWAY|GW][,] [GROUPID|GID][,] [HWCAP][,] [HWREQ][,] [HYPERREL|HYREL][,] [HYPERRELRUN|HYRELRUN][,] [IDS|ID|I][,] [IFNAME][,] [INDEX][,] [LABEL|L][,] [MAC|M][,] [NETMASK][,] [NETNAME][,] [OS|O][,] [OSREL][,] [PLATFORM|PFORM][,] [PM|HOST|H][,] [PNAME|P][,] [RELAY][,] [RELOCCAP][,] [SERIALNUMBER|SERNO][,] [SERVERACCESS|SPORT|S][,] [SSHPORT][,] [STACKCAP|SCAP][,] [STACKREQ|SREQ][,] [TCP|T][,] [TYPE|STYPE|ST][,] [USERSTRING|USTRG][,] [USERID|UID][,] [UUID|U][,] [VCPU][,] [VERSION|VERNO|VER][,] [VMSTATE|VSTAT][,] [VNCBASE][,] [VNCDISPLAY|DISP][,] [VRAM][,] ) [TITLE|TITLEIDX|TITLEIDXASC][,] [MACHINE|MAXKEY][,] ) | (TAB_GEN|TAB)[:<tab-args>] | (REC_GEN|REC)[:<tab-args>] | (SPEC_GEN|SPEC)[:<tab-args>] | (XML_GEN|XML)[:<tab-args>] )[,] [IP|DNS|D][,] [SORT[:ALL|A|UNIQUE|U|REVERSE|R|EACH |[([0-9][0-9][0-9]|[0-9][0-9]|[0-9])] [%]][,] ] [-p <db-directory-path-list>] [-r] [-s] [-S <BasicDataManagementSupport> =:( CONTENTGROUP |LISTALL |LIST |LISTDB |MEMBERSDB |LISTGROUP |MEMBERSGROUP([2345678])|([678]u)[:<groups-list>] ) ] [-T <type-list>] [-V] [-R <runtime states> =:( [MARK|(REVERSE|R|-),](PING|SSH)[,(PM|VM)] ) ] [-X] <awk-regexpr>[ ( AND |E:<#field0>:<#field1> |F:<#field>:<string-literal> |NOT |OR |<awk-regexpr> ) ]
ctys-vhost is the basic address resolution interface for runtime execution of commands based on ctys addressing (Address-Resolution) . The main task of this tool is to support a scripting interface for the functional link between VMs and PM, and their contained OS. Even though a considerable amount of functionality is provided, the user interface is designed in a manner for simplified application of the common daily tasks. The basic queries just require actually a few arguments, e.g.
ctys lab02 tst320
displays by default the record for the VM 'tst320' registed for execution on the PM 'lab03'. The following query
ctys lab02 tst32
displays on the PM 'lab02' all VMs with the pattern match '*tst32*', whereas
ctys lab02 tst3.0
displaye the pattern match '*tst3.0*' - where the '.' matches any character. When required a more detailed set of constraints on a query could be applied as required.
The 'ctys-vhost' utility combines functionality of:
The basic similarity to the UNIX 'host' function for name resolution to virtual machines is expanded by several features, which take into account the roaming of VMs and thus changing their actual execution path within a so called 'execution stack' assembled by PMs, VMs, and HOSTs. This implies some dynamic data handling due to frequently changing 'physical containment positions' of roaming VMs. Therefore a basic load-balancing is included, which is currently still a pre-release.
Due to targeted simplicity and efficiency the address resolution is based on a combination of RECORD based and unstructured flat pattern matching by regular expressions only. Even though the record is structured by fields as given by '-a ENUMERATE=MACHINE' , the match itself could be performed quite reliable and performant as a simple regexpr given by an awk-match string expression to be applied to RECORD-STRING, whereas the output is still displayed on the level of records and fields. Combined multiple-regexprs are supported and are iteratively applied on the intermediary results of each step. The set of regular expression could be either combined by AND or OR operators, where AND is the default behaviour. This opens for more complex queries. The behaviour of the operator preference is described in detail within the following ARGUMENTS section. The filtering of the output as requested by the '-o' option is performed on the final set of resulting records only. This could be applied for example, when a table display has to be sorted, where the option '-o SORT:1' sorts the resulting table by the first column. The call with the option '-o sort:1,reverse' results in reverse sorting of the first column.
This simplified pattern matching approach is applicable when the given set of search attributes result in a unique match. When ambiguity occurs due to multiple matching records, the set of records are handled as defined by the '-M' option. Ambiguity occurs frequently when using a reduced set of selection attributes on a database for VMs in a networked environment. This is due to the frequently desired availability of multiple redundant acces targets for the physically identical VM. Either the specification of additional attributes, or the application of load-distribution policies will resolve ambiguity. In current release a basic COST option '-C' extends the '-M' option for some usable load distribution within execution-groups.
In most other cases uniqueness should be given, e.g. the key UUID is defined to be applied as unique, but could be tampered unintentionally e.g. by co-allocated backups. For avoiding of backup-access the '-M first' option might be helpful, when the name of the backup is an extension. The more reliable approach here is to set the VMSTATE to BACKUP and using the VMSTAT filtering attribute for the ENUMERATE action when collecting the inventory data.
The current implementation enables the management of up to multiple thousands of VMs with the given simplicity with single-query resposes in the range of about 0.6-0.8 seconds(measured, but ofcourse depends on machine etc.). One of the next versions will additionally support LDAP based nameservices, targeting an enterprise environment with a number of almost an unlimited amount of distributed services to be managed.
When a mapping table MAC-IPAddress-HostName - here named as 'macmap-DB' - is present this will be used for open mappings which are not configured within the VM configuration files. Particularly any IP or PM/Hostname address for a given MAC-Address is resolved when not present within the enum-DB.
The address resolution is performed by the following steps (ctys-vhost data) :
The following list shows some examples of using ctys-vhost as an interactive query tool.
Additionally to the flat-matching by simple regular expressions some
additional keywords are defined.
These are
AND,
OR,
NOT,
E:, and
F:,
described within the
section related to the ARGUMENTS.
.
<DB sources>= ( OFF| CLEARTMP| CLEARALL| GROUPS| KEEPALL| LIST| LISTARGETS| LISTGROUPS| MEMBERSDB| MACMAPONLY| MACMAP| REBUILDCACHE )This group of keywords controls the runtime behaviour, which has an impact to the overall performance.
<input-list>=[CTYSADDRESS|CTYS]
ctys-vhost <in-out-options> <arg1> <arg2> <agr3>For the display of the actual contents of a specific intermediate step in addition to it's draft performance-overview, just drop all following filters/arguments from the call, what will display the requested result as final. This result is identical to the covered intermediate result when using it within a chained set of filters.
DEFAULT=CTYS_NETACCOUNT(DEFAULT->USER)
<result-set-output-reduction> =(FIRST|LAST|ALL|COMPLEMENT|SORT|USORT|UNIQUE)
TAB_CTYS_VHOST_DEFAULTThis table could be customized as required, but should be handeled carefully.
<output-list>= ( ( ( [ARCH][,] [ACCELERATOR|ACCEL][,] [CATEGORY|CAT][,] [CONTEXTSTRING|CSTRG][,] [CPORT|VNCPORT][,] [CTYSADDRESS|CTYS][,] [CTYSRELEASE][,] [DIST][,] [DISTREL][,] [EXECLOCATION][,] [EXEPATH|EXEP][,] [GATEWAY|GW][,] [GROUPID|GID][,] [HWCAP][,] [HWREQ][,] [HYPERREL|HYREL][,] [HYPERRELRUN|HYRELRUN][,] [IDS|ID|I][,] [IFNAME][,] [INDEX][,] [LABEL|L][,] [MAC|M][,] [NETMASK][,] [NETNAME][,] [OS|O][,] [OSREL][,] [PLATFORM|PFORM][,] [PM|HOST|H][,] [PNAME|P][,] [RELAY][,] [RELOCCAP][,] [SERIALNUMBER|SERNO][,] [SERVERACCESS|SPORT|S][,] [SSHPORT][,] [STACKCAP|SCAP][,] [STACKREQ|SREQ][,] [TCP|T][,] [TYPE|STYPE|ST][,] [USERSTRING|USTRG][,] [USERID|UID][,] [UUID|U][,] [VCPU][,] [VERSION|VERNO|VER][,] [VMSTATE|VSTAT][,] [VNCBASE][,] [VNCDISPLAY|DISP][,] [VRAM][,] ) [TITLE|TITLEIDX|TITLEIDXASC][,] [MACHINE|MAXKEY][,] ) | TAB_GEN[:<tab-args>] | REC_GEN|REC[:<tab-args>] | SPEC_GEN|SPEC[:<tab-args>] | XML_GEN|XML[:<tab-args>] ) [IP|DNS][,] [,SORT[:<sort-args>]]
<sort-args>=[ALL|A|EACH][%(UNIQUE|U)][%(REVERSE|R)][%<sort-key>]
ctys-vhost -o cport,l -M all '59\[0-9\]\[0-9\]'Where a standard baseport of 5900 is assumed.
<BasicManagementSupport>= ( LISTALL |LIST |LISTDB |MEMBERSDB |LISTGROUP[:<groups-list>] |MEMBERSGROUP([2345678])|([678]u)[:<groups-list>] ) <groups-list>:=<group>[,<groups-list>]The following keywords may be applied.
LIST = LISTDB + LISTGROUP
" <size> <#lrec>/<#incs> <#srec> <group>"
<group-list> =: <group-name>[%<group-list>]Two types of storage are shown:
<runtime states> =[MARK|(REVERSE|R|-),]PING|SSH[,PM|VM]
.
Use "-I 2" option for some performance analysis of order dependency for multiple-selection queries.
The next point to be aware of is the order of precedence for the logical operators AND and OR. This is evaluated from left-to-right, the precedence is inverted. This means, that lower precedences are grouped together by virtual braces. This academic sounding remark offers tremendous positive practical impacts. First the theory - as a result the following rules are identical.
x AND a OR b OR c
is equal to
a OR b OR c AND x
These could be written as
x AND a OR x AND b OR x AND c
Current version does not support braces, but if - the previous could be written as
x AND ( a OR b OR c )
The next consequence is the repetition of the rules when a higher valued operator AND follows a lower valued group.
x AND a OR b OR c AND y
This again could be written as
x AND ( a OR b OR c ) AND y
Or better as
( x AND ( a OR b OR c ) ) AND y
This is consequently equal to
y AND ( x AND ( a OR b OR c ) )
Which could be written as
y AND x AND ( a OR b OR c )
The main advance of the previous theory is the tremendous simplification and size-reduction of
vast powerful and fast scanners.
Thus within just a few lines a complete scanner with a recursive logic could be implemented as
present within the 'ctys-vhost' command.
The call
ctys-vhost -o sort:1 \ tst00 and not f:2:PM and tst103 or tst00 or tst00
or
ctys-vhost -o sort:1 \ not f:2:PM and tst103 or tst00 or tst00 and tst00
selects the following set
label |styp|accel|dist|distrel|os |osrel|PM |if|TCP ------+----+-----+----+-------+-----+-----+-----------+--+------------ tst003|VMW | |SuSE|9.3 |Linux|2.6 |delphi.soho|0 |172.20.2.133 tst003|VMW | |SuSE|9.3 |Linux|2.6 |delphi.soho|0 |172.20.2.133 tst005|VBOX|HVM | | |Windo| |lab02.soho | | tst005|VBOX|HVM | | |Windo| |lab02.soho | | tst005|VBOX| | | | | |lab02.soho | | tst005|VBOX| | | | | |lab02.soho | |
from a databse with 837 entries.
The actual queries could be visualized for analysis porposes.
Therefore the intermediate metadata and the resulting matched records are displayed by the
option '-I 2'.
The first call produces the output
time ctys-vhost -I 2 -o sort:1 \ tst00 and not f:2:PM and tst103 or tst00 or tst00 START R-Methods CHECK =/homen/acue/.ctys/db/default START R/W-Methods RM RTCACHE =/tmp/ctys.acue/ctys-vhost.20101106184019.cdb QUERY:cacheDB repetitive:"tst00" ....!!!!!X!!!!!!!!.x.........x.........x.........x50 .........x.........x.........x.........x.........x100 .........x.........x.........x.........x.........x150 .........x.........x.........x....!!!!!X!!!!!!!!.x200 .........x.........x.........x.........x.........x250 .........x.........x.........x.........x.........x300 .........x.........x..!!!!!!!X.........x.........x350 .........x.........x.........x.........x.........x400 .........x.........x.........x.........x.........x450 !........x.........x.........x.........x.........x500 !!!!!!!!!X!!!!!!!!!X!........x.........x.........x550 .........x.........x.........x.........x.........x600 .........x.........x.........x.........x.........x650 .......!!x.........x.........x.........x.........x700 .........x.........x.........x!........x.........x750 .........x.........x.........x!!!!!!!..x.........x800 .........x.........x.........x..... match=68 of total=835 QUERY:cacheDB repetitive:"f:2:PM" .........x.........x.........x......!..x.........x50 ........!X........ match=3 of total=68 QUERY:cacheDB repetitive:"tst103" ... match=0 of total=3 QUERY:cacheDB repetitive:"tst00" !!! match=3 of total=3 QUERY:cacheDB repetitive:"" !!! match=3 of total=3 QUERY:cacheDB repetitive:"tst00" !!! match=3 of total=3 QUERY:cacheDB repetitive:"" !!!!!! match=6 of total=6 !!!!!! match=6 of total=6 label |styp|accel|dist|distrel|os |osrel|PM |if|TCP ------+----+-----+----+-------+-----+-----+-----------+--+------------ tst003|VMW | |SuSE|9.3 |Linux|2.6 |delphi.soho|0 |172.20.2.133 tst003|VMW | |SuSE|9.3 |Linux|2.6 |delphi.soho|0 |172.20.2.133 tst005|VBOX|HVM | | |Windo| |lab02.soho | | tst005|VBOX|HVM | | |Windo| |lab02.soho | | tst005|VBOX| | | | | |lab02.soho | | tst005|VBOX| | | | | |lab02.soho | | real 0m1.176s user 0m0.696s sys 0m0.532s
which requires less steps due to early AND reduction of the metadata.
The second call requires much more steps, this is due to the late AND reduction of the metadata, which
causes any OR part to be AND checked on the whole initial set.
time ctys-vhost -I 2 -o sort:1 \ not f:2:PM and tst103 or tst00 or tst00 and tst00 START R-Methods CHECK =/homen/acue/.ctys/db/default START R/W-Methods RM RTCACHE =/tmp/ctys.acue/ctys-vhost.20101106184201.cdb QUERY:cacheDB repetitive:"f:2:PM" .........x.........x.........x.........x.........x50 .........x.........x.........x.........x..!!!!!!!X100 !!!!!!!!!X!!!!!!!!!X!!!!!!!!!X!!!!!!!!!X!!!!!!!!!X150 !!!!!!!!!X!!!!!!!!!X!!!!!!!!!X!!.......x.........x200 .........x.........x.........x.........x.........x250 .........x.........x..!!!!!!!X!!!!!!!!!X!!!!!!!!!X300 !!!!!!!!!X!!!!!!!..x.........x.........x.........x350 .........x.........x.........x.........x...!!!!!!X400 !!!!!!!!!X!!!!!!!!!X!!!!!!!!!X!!!!!!!!!X!!!!!!!!!X450 !!!!!!!!!X!!!!!!!!!X!!!!!!!!!X!!!!!!!!!X!!.......x500 .........x.........x.........x.........x.........x550 .........x.........x.........x.........x.........x600 .........x.........x.........x.!!!!!!!!X!!!!!!!!!X650 !!!!!!!!!X!!!!!!!!!X!!!!!!!!!X!!!!!!!!!X!!!!!!!!!X700 !!!!!!!!!X!!!!!!!!!X!!!!.....x.........x.........x750 ......!!!X!!!!!!!!!X!!!!!!!!.x.........x.........x800 .........x.........x.........x..... match=349 of total=835 QUERY:cacheDB repetitive:"tst103" .........x.........x.........x.........x.........x50 .........x.........x.........x.........x.........x100 .........x.........x.........x.........x.........x150 .........x.........x.........x.........x...!.....x200 .........x.........x.........x.........x.........x250 .........x.........x.........x.........x.........x300 .........x.........x.........x.........x......... match=1 of total=349 QUERY:cacheDB repetitive:"tst00" .........x.........x.........x.........x.........x50 .........x.........x.........x.........x.........x100 .........x.........x.........x.........x.........x150 .........x.........x.........x.........x..!......x200 .........x.........x.........x.........x.........x250 .........x!!.......x.........x.........x.........x300 .........x.........x.........x.........x......... match=3 of total=349 QUERY:cacheDB repetitive:"tst00" .........x.........x.........x.........x.........x50 .........x.........x.........x.........x.........x100 .........x.........x.........x.........x.........x150 .........x.........x.........x.........x..!......x200 .........x.........x.........x.........x.........x250 .........x!!.......x.........x.........x.........x300 .........x.........x.........x.........x......... match=3 of total=349 QUERY:cacheDB repetitive:"tst00" .!!!!!! match=6 of total=7 QUERY:cacheDB repetitive:"" !!!!!! match=6 of total=6 !!!!!! match=6 of total=6 label |styp|accel|dist|distrel|os |osrel|PM |if|TCP ------+----+-----+----+-------+-----+-----+-----------+--+------------ tst003|VMW | |SuSE|9.3 |Linux|2.6 |delphi.soho|0 |172.20.2.133 tst003|VMW | |SuSE|9.3 |Linux|2.6 |delphi.soho|0 |172.20.2.133 tst005|VBOX|HVM | | |Windo| |lab02.soho | | tst005|VBOX|HVM | | |Windo| |lab02.soho | | tst005|VBOX| | | | | |lab02.soho | | tst005|VBOX| | | | | |lab02.soho | | real 0m1.256s user 0m0.836s sys 0m0.480s
Another effect which could be seen here is the almost equal processing duration, which
shows the startup and display overhead as the dominating factor.
The times are in the same range for half of the data as shown for 835 entries.
The seemingly double-display is due to the multiple OR in this inverted-precedence semantics, which is resulting in addtional display.
The seemingly double-display is due to the hidden display of the owner, here one is the user 'root.root', the other is 'acue.ldapusers'.
For the following rules and operators the search-and-match strings are case sensitive, the operators are not.
.
Written and maintained by Arno-Can Uestuensoez:
Maintenance: | <<acue_sf1 (a) sourceforge net>> |
Homepage: | <https://arnocan.wordpress.com> |
Sourceforge.net: | <http://sourceforge.net/projects/ctys> |
Project moved from Berlios.de to OSDN.net: | <https://osdn.net/projects/ctys> |
Commercial: | <https://arnocan.wordpress.com> |
Copyright (C) 2008, 2009, 2010 Ingenieurbuero Arno-Can Uestuensoez
For BASE package following licenses apply,
This document is part of the DOC package,
For additional information refer to enclosed Releasenotes and License files.