TMQL Issues
There is a live version of the TMQL Issues document that is maintained by Robert Barta. Please feel free to give feedback directly to Robert Barta or the TMQL mailing list.
TMQL Issue: Is everything a 'thing'
Impact on Language: medium
Background
TMDM defines the concept 'subject', but it says nowhere that all things
in the map (topics and associations, for instance) are instances of
'subjects'. And it also does not say that all types are subtypes of
'subject'.
The problem is that TMQL should offer users a 'catchall' concept, such
that they can express a 'dont care semantics':
select $person
where is-employed-by (* : tbl, organisation: $o) &
$o isa ....
And the TMQL semantics of course also needs access to 'all the things in
a map'.
What does this * amount to now? Is it identical to tm:subject? And are
there specialized terms for (a) topics, (b) associations, (c)
occurrences?
Structured Discussion
? Does tm:subject return all items in the map?
+ tm:subject is a great placeholder
- TMDM does not say it (or does it?)
? How do topics, associations, .... relate then to tm:subject?
TMQL Issue: Intuitive Semantics of 'false'
Impact on Language: medium
Background
'true' and 'false' have two meanings. One is that they are just values of a data type
boolean, in the same way as 3, 3.14 or "sunshine" are values of other types.
So in this sense
select $person
where
exists 3.14
will return all persons as there always exists the constant 3.14. But what about
select $person
where
exists false
Also 'false' exists always, but most users would think that the above is equivalent with
select $person
where
false
which would - in fact - return nothing.
So there is a constant 'false' outside boolean expression and that behaves like a value. And
then there is 'false' inside boolean.
Structured Discussion
? introduce a constant NULL which renders always an empty sequence.
? is this related to xsd:AnyType?
- NULL is nothing else than ()
+ would cleanly separate the constant 'false' from the 'untrue' NULL
TMQL Issue: Functions process sequences
Impact on Language: medium
Background
Functions in TMQL are - as usual - invoked with parameters, such as in
if math:sqrt ($person / shoesize) < 10
then
"big square foot"
Parameter are usually computed and - because of the nature of topic maps
- the number of results may vary. What if a person does not have a
shoesize at all, or 100eds of them?
One way to deal with that is to generalize all functions being able to
process whole sequences. If a person has shoesizes 2, 3, and 4, then
math:sqr ( $person / shoesize )
will return a sequence 4, 9, 16. That is of course cheap to implement.
This procedure would also be consistent with the case that we had
several parameters, each potentially generated a sequence of values:
terrorism:arrest-them-all ($person / name, $country / name, $pretext / name)
The tuple expression may generate tuple sequences with varying length,
depending on how many names exist for the things above.
Structured Discussion
? functions operate always on tuple sequences
+ straight-forward semantics
+ formal semantics already defines functions so
+ easy to implement and also fast in implementation as calls can be optimized away
- slightly unsual semantics for normal engineers, they are not used to sequence processing
- Python, Ruby, Perl and Haskell all have list comprehension, so what?
TMQL Issue: Datatype Awareness
Impact on Language: medium
Background
At the moment, TMQL treats atomic data values as exactly this: atomic. This implies that
the following is NOT possible:
select $person / name
where
$person / shoesize >> types == xsd:float
[ find all persons which have as shoesize a FLOAT value ]
Structured Discussion
? Make TMQL data type aware?
+ makes it easy for these kind of queries which navigate to the type of a value
- it contradicts TMDM which stays silent on that matter
- can be done with a function tm:datatype ($person / shoesize)
TMQL Issue: intermediate types
Impact on Language: low
Background
Some approaches for a TM query language have introduced language features to extract all or
a particular part of the types of a topic, usually via a distance operand:
types ( cat , 3 ) # get the type, its super types, and theirs
types ( cat, * ) # get all of them
TMQL does not support this, mainly because using distances in the type structure in a query
might be brittle as the type structure might be refined later.
Structured Discussion
? Add language feature (or function) for intermediate types
+ some people will use it
- may lead to brittle queries
- we are not here to educate the world
TMQL Issue: Add inverse reification shorthand
Impact on Language: low
Background
TMQL allows to follow the 'reification' axis in both directions. It leads from a topic to
the item it reifies or the other way 'round. For the forward direction there is already a
shortcut ~~> defined.
When in CTM a reification is declared, then there also a symbol has to be used, one
candidate being <~~. The question is now whether TMQL should have this as another
shortcut:
<~~ ==> << reifier
Structured Discussion
? Add <~~ ==> << reifier as short cut
+ little cost
+ alignment with CTM
- little used in TMQL itself
TMQL Issue: item identifiers for items
Impact on Language: low
Background
At the moment, TMQL does not allow to retrieve item identifiers within a
query. It is still possible to write
select $person
where
$person isa person
and that will produce a sequence of items; whether these are passed as
items or item identifiers back to the calling application, TMQL does not
say. Also when items are used in FLWR expressions, sometimes their id is
used, depending on the context where they are embedded.
What does not work (yet) is to say
select $person >> id
mainly because it is unclear whether there is a use case for it.
Inside a query, item identifiers could be used for comparing
where
$person >> id == 'whatever'
but this can be expressed less baroquely as
where
$person == whatever
What could be interesting is to learn about the item identifiers of associations and
characteristics
// is-employed-at >> id
or to create a string from the item identifier
select "this:is-all-weird#" + $person >> id
Structured Discussion
? Should a new navigation be added to retrieve item identifiers
+ does not disturb current navigation structure, returns a string
- is there a use case for it?
TMQL Issue: RegExp operator in language
Impact on Language: low
Background
Sometimes it would be convenient to have a Perl-like regexp operator
inside the language, such as
select $person
where $person is person
& $person / name =~ /^Tim/i
Structured Discussion
? should TMQL have a binary regexp operator
? what would be the syntax
- can be done with functions as well
? what regexp exactly (XPath, fn:matches)?
+ better readability
TMQL Issue: alias 'except' for --
Impact on Language: low
Background
The operator '--' subtracts two sequences, such as in
// person -- // evildoing-evildoers
Should this also be allowed (all things about the person except the
shoesize):
$p / * except $p / shoesize
Structured Discussion
? Should TMQL alias '--' to except ?
+ human-friendly syntax
- very little addition
== RESOLVED ISSUES =============================================================
TMQL Issue: Removal of EVERY clause
RESOLVED
Impact on Language: medium
Background
In the Leipzig meeting it was discussed that the EVERY clause is only a syntactic variation
of the SOME clause. Every EVERY can be transformed into an equivalent form using SOME:
every $p in // person
satisfies $p / born
==> not
some $p in // person
satisfies not $p / born
And hence, EVERY can be removed.
Following this argumentation much of the language can be removed. Also SOME is just a
variation of FLWR:
some $p in // person
satifies $p / born
==>
for $p in // person
where
$p / born
return
$p
And FLWR expressions can be transformed into path expression, etc.
Structured Discussion
? Removal of EVERY clause
+ minimally smaller language
- users have to twist their brains to get the NOT right
! Oslo 2007: consensus: rejected
? Removal of SOME clause
+ it is just a convenience
- the symmetry is something which people expect
! Oslo 2007: vote (1/many) rejected
TMQL Issue: Refactorization of Standard-Parts
RESOLVED
Impact on Language: high
Background
TMQL contains a number of sections which can be argued to actually belong into other TM
standard documents. They have only be added to make TMQL work.
- Atoms and Identifiers -> CTM
- Navigation -> TMDM
- Environmental Clause -> CTM
- Predicates -> TMCL/CTM
- Ontologies -> TMCL/CTM
- Predef Types & Functions -> CTM/TMCL
If these would be factored out, then the TMQL specification would be reduced by roughly 25%
.
Structured Discussion
? Refactor TMQL standards part into TMDM, CTM and TMCL?
+ cleaner specification landscape
- TMDM is put in stone already
+ CTM and TMCL are close to finalization, do it now or never
! Oslo 2007: refine it as below
? Refactor atom handling into CTM
= which atoms are there, how do they look like
= would mean that this is defined in CTM
= would mean that TMQL is just reusing the definitions
! Oslo 2007: accepted by consensus
? Refactor inferencing
= at the moment inferencing is "predicates", i.e. associations only
not general (inferring values for occurrences, for example)
= taxonometric reasoning is implicitly defined by TMDM, should stay in TMQL
= means: refactor into what?
= new language for ontology definition (TIMBL, TOWL), or
- may take time to jump-start
- but Patrick says, this can be relatively fast
= squeeze it into TMCL?
- constraints and ontological info not the same
- still: it means, that TMQL must ___ALLOW___ to specify onto info inline
+ huge benefit: TMQL is itself inferencing-agnostic and is operating on
virtual map
+ different people have different needs
! Oslo 2007: accepted by consensus
? Refactoring ontology
= means how to define that something is a TMish resource (can be anything)
+ yes, this does not affect TMQL at all
- but there still needs a way to 'access' ontologies (as topics)
= means a 'prefix' mechanics
= AND!!!! means that if it is a topic it can be treated as such
= can go into TIMBL/TMOL
! Oslo 2007: accepted by consensus
? Refactor 'Navigation'
= Navigation is effectively the expectation a TMQL processor
has about the map it navigates through.
? refactor where to?
= into TMDM annex?
- but this may be difficult as TMDM is '60.60'
= into separate document?
- does this have a value by itself?
+ maybe only for architects
! Oslo 2007: rejected by vote
= keep navigation axes semantics inside TMQL
TMQL Issue: Quantified Quantifiers
RESOLVED
Impact on Language: medium
Background
At the moment quantifiers in TMQL are only of a EVERY or SOME nature, there is no way to say
"give me all hands with at least 5 fingers".
It is believed, that such a feature is required by TMCL and since TMCL will define its
modelling patterns with TMQL, there is some issue here.
So the idea would be to allow
select $person
where
at least 5 $finger in $person <- * -> finger
satisfies
$finger / status == "functional"
or
select $person
where
exists at most 2 $finger in $person <- * -> finger
This functionality can be mapped into the existing TMQL, though, although this may be
cumbersome to do it manually. The first would be transformed into
# there is one finger, satisfying this
# and another finger, satisfying this
# and another
#
# and the fifth one
The other would be transformed into the logic equivalent
"the person has not at least 2 fingers satisfying".
Structured Discussion
? Should TMQL get quantifiers AT MOST integer, AT LEAST integer?
+ can be mapped into canonical TMQL by the processor
- can be expensive to compute
+ TMCL would profit from that
- a function fn:count (....) >= 10 has the same effect, so why introduce new keywords
- not true: fn:count has to produce ALL results, count them and then do compare
= an 'at least' expresses more what a user needs
+ can be used for optimization
! Oslo 2007: vote (7 yes, 1 no)
= editors will present a proposal in the next draft
TMQL Issue: Allow variables as role type
RESOLVED
Impact on Language: medium
Background
At the moment, TMQL allows only constants for roles when navigating to or from an
association:
....
where
$person <- employee -> employer == big-bad-corp
It would be possible to generalize this and allow PEs and with them variables as roles
$person <- $whatever_role [ ^ is-employed-at ]
As path expressions and association predicates are closely connected, this also means that
variables for roles are possible there too:
is-employed-at ($whatever_role : $person, ...)
This can also be extended to 'variables as assoc type':
select $person / name,
where
$assoc_type ($whatever: $person, ....)
select $person, $foo
where
$person / $foo
Structured Discussion
? should full path expressions be allowed for role (types)
- are there use cases for this? Not in the 'official' list
+ should not be too expensive to implement
+ no computational complexity added
- makes it impossible to use common index structures, may be slow
! Oslo 2007, rejected by consensus
? should variables be allowed for role types?
- impedes optimization (use of indices)
! Oslo 2007: weak acceptance
? should variables be allowed for assoc types?
- impedes optimization (use of indices)
! Oslo 2007: weak acceptance
TMQL Issue: Elimination of the concept 'characteristics'
RESOLVED
Impact on Language: medium
Background
TMQL is using the term 'characteristics' as subsumption for topic names
and topic occurrences. This is in deviation to an earlier interpretation
which also included topic roles. TMDM does not use the term
'characteristics' anymore.
The reason for re-introducing it in TMQL was that that way names and
occurrences could be treated via one syntactic structure. It would
equally be possible to break names and occurrences in two and introduced
identical navigation mechanism for this.
Structured Discussion
? Should the concept of characteristics be broken up into 'names' and 'occurrences'?
+ alignment with TMDM in this respect
- pretty useless duplication of navigation, it is the same
+ 'characteristics' had a different meaning in the past
! Oslo 2007: consensus, better align it with TMDM
TMQL Issue: Error compatibility
RESOLVED
Impact on Language: high
Background
At the moment, the TMQL standard does not detail the errors which can occur during the
static/dynamic analysis; and it also does not give the errorenous situations a name.
From a language perspective this is not necessary, but OTOH, it reduces compatibility
between TMQL processors as one application has to expect potentially different sets of
exceptions.
An example of a transient error would be
select $p
from http://www.topicmaps-are-us.com/ ~ ~~>
where
$p isa person
Structured Discussion
? Should TMQL specify the mechanism?
= i.e. exception propagation, or error codes?
- this does preempt implementation styles
! Oslo 2007: consensus, maybe better not
? Should the TMQL standard name all error situations?
+ higher compatibility between TMQL implementations
- less freedom for implementors
- makes the standard take longer
- ok, maybe a small bit
- some conditions may be hard to differentiate, depending on the implementation strategy
- compatibility on the error-level would be API compatibility
! Oslo 2007: rejected by vote (2/many)
= no list
? Should TMQL name only a top-level?
= error in analysis, error in evaluation
= persistent error, transient error (Geir Ove Gronmo)
+ somewhat a middleground
! Oslo 2007: consensus: then only 'good' vs. 'bad'
TMQL Issue: XML generation, XML subset
RESOLVED
Impact on Language: high
Background
TMQL allows to generate XML content.
return
<terrorists>{
for $person in // person
where
soundex ($person / name , "isimi-bin-lidin")
return
<evil-evildoer>{$person / name}</evil-evildoer>
}</terrorists>
In that, XML content is _embedded_ into the TMQL expression, this is NOT just a text
template which is expanded. The advantage is that processors can included this in the
optimization process.
The XML allowed here at the moment does not reflect XML in all its beauty. So, for instance,
TMQL does not support CDATA, processing instructions or XML namespaces. The reasoning being
that all this can be much more effectively handled with XSLT.
Other ideas are to use an XSLTish syntax additionally for some flexibility:
<tmql:comment>
sdfsfdsfd
</tmql:comment>
<tmql:element name="whatever">
<tmql:attribute name="aname">avalue</tmql:attribute>
content
</tmql:element>
Structured Discussion
=> Issue: TMQL native XML support
? TMQL should support 100% of XML?
- quite expensive to implement
- missing pieces can be added with an XSLT processor much more effective
+ completeness is a beautiful thing
! Oslo 2007: what do we *really* need?
CDATA NO
PI NO
namespace YES
DOCTYPE NO
comments NO
variable element names YES
variable attribute names YES
TMQL Issue: native XML support
RESOLVED
Impact on Language: High
Background
In principle TMQL could do without any XML support. The only thing to be specified is that
results should be generated in a fix, defined XML structure.
The downside of this is that SELECT, FLWR and path expressions only can generate tuple
sequences and that the XML format only can contain exactly these.
Structured Discussion
? TMQL should support 0% of XML
+ TMQL syntax becomes smaller by 6 rules
+ implementation is a bit simpler
- results cannot be free, deeply nested XML, but only flat
- one always will need XSLT to beat the results into proper shape
! Oslo 2007: implicit rejection, see issue native-XML-support
TMQL Issue: predefined data types
RESOLVED
Impact on Language: very high
Background
At the moment, TMQL has a rather arbitrary choice of primitive data types: integer, URIs,
decimals, datetimes, most of them cloned from the XML schema data types
http://www.w3.org/TR/xmlschema-2/datatypes.html#typesystem
The question is which of these should be in the final standard. Maybe the following (lheuer)?
- One of the floating point numbers
- one integer datatype
- xs:string
- xs:anyURI
- xs:boolean
- xs:date
- xs:dateTime ???
- xs:time ???
And what about an 'undefined' data type to express that the results is
_undefined_?
select $p / shoesize || undefined
where $p isa person
Maybe also aggregation functions?
http://www.w3.org/TR/xpath-functions/#aggregate-functions
Another alternative is not to define _any_ primitive data type and leave that to implementations. The only
thing to define in the standard are minimal requirements each data type has to satisfy:
serialisation rules (text to value, and back)
ordering (the meaning of <= operator)
Still, in TMQL, the binary operators +, -, *, / and the unary operator - can be left between value expressions.
Implementations can the detail how these operators are then be interpreted for particular data types.
Structured Discussion
? should all types from XSD be understood?
+ very rich collection, strong big-vendor support
+ constants can be written 'naturally' (3 instead of "3"^xsd:integer)
=> also all XPath 2.0 functions and operators have to be supported?
- very expensive to produce for small developers
- supported data types and supported functions are not strictly connected
! Oslo 2007: rejected by consensus
=> see issue 'refactoring': all predefined datatypes are named by CTM
? should NO type be native?
+ postpones decision to implementation
- reduces the compatibility between implementations
! Oslo 2007: rejected by consensus
? should there be a predefined constant 'undefined'?
+ allows to express 'undefined' value
! Oslo 2007: weak acceptance
TMQL Issue: occurrence type implicit subclassing
RESOLVED
Impact on Language: high
Background
## Comment: NOT A TMQL, but a CTM issue
When someone writes in CTM (modulo syntax details)
tbl isa person
! name: Tim Berners Lee
! nickname: TBL
homepage: http://www.bigtim.com
then what it implicitely means is that nickname is a subclass of a name and homepage is a
subclass of an occurrence.
TMDM never spells that out explicitly, because it is irrelevant there. Only in serialization
syntaxes like XTM or CTM this has to be defined.
Structured Discussion
? should an occurrence type be implicitly a subtype of 'occurrence'?
+ this is what people will expect
= also that a nickname is a special name
! Oslo 2007 resolution: accepted
? where should this fact be standardized?
= TMDM
- even an annex is difficult to add
! Oslo 2007: rejected by consensus
= CTM
- why CTM and not XTM (et.al)?
+ because it would be at least one place to get it right?
! Oslo 2007: rejected
= put it into TMDM -> TMRM mapping (LarsM)
! Oslo 2007: accepted by consensus
TMQL Issue: TM paradigmatic functions, predicates, templates, ontologies
RESOLVED
Impact on Language: very high
Background
A function is a systematic functional dependency over particular sets of values. The age of
a person, for example, is functionally dependent on (a) the person's birthdate and (b) the
current time. Similarily, a predicate is a constraint on a constellation of particular
values, and items within a topic map.
Accordingly, both are expressing additional knowledge about a problem domain. As the
modelling of an application domain is usually done via an ontology definition language, one
can argue that functions and predicates should NOT be part of TMQL, which is meants as a
data access language.
Structured Discussion
? Should Functions and Predicates be included into TMQL
- both are ontological information, should go into an ontology language
- yes, but TMCL will not offer them, neither does CTM, ...
+ they are EXTREMELY useful to organize the query, uhm knowledge
- SPARQL does not have function declarations or predicate declarations
! Oslo 2007: no, by consensus
=> all this should be hosted inside a TM 'ontology language'
=> still, TMQL must now allow statements made in this "TMOL" language
TMQL Issue: Functions and Predicates as first-class topics
RESOLVED
Impact on Language: very high
Background
If it is accepted that TMQL should contain features to define local functions and predicates
(which are specialized functions), the question is then how this is integrated conceptually
and syntactically into TMQL.
One option is to use a convential syntax, something like
function nr_employees (organisation: $o)
return
fn:length ( $o <- employer )
'function' and 'return' would become TMQL keywords connecting together the function name
with the function body. The function body - by its nature - will always be a TMQL
expression.
The parameter profile of the function - here after the function name inside a () pair -
would specify which variables are to be treated as constants, i.e. those where the caller
will provide values for at invocation time. Additionally, a type (here organisation) can
provide additional information to the TMQL processor which it may (or may not) use.
Technically it would be sufficient to write
function nr_employees ($o)
return
fn:length ( $o <- employer )
or even
function nr_employees
return
fn:length ( $o <- employer )
if a rule is introduced that all unbound variables (those not explicitly quantified with
FOR, SOME or EVERY) become a parameter. That rule is implicit already as it would be
redundant to have a parameter list and a list of unbound variables. It would then be an
error if the two lists would be different.
A similar structure and convention can be introduced for predicates
predicate is_NGO
where
not $o <- part -> whole == // government
That checks whether an organisation is part of anything which itself is an instance of a
government (the == comparison is always interpreted existentially in TMQL!).
If it is also accepted that functions and predicates are actually ontological information
then also a more TMish syntax can be chosen, especially since the only task is to bind an
expression to a name:
nr_employees isa tmql:function
tmql:return : {
fn:length ( $o <- employer)
}
Hereby CTM can be used (AsTMa= here only for demonstration). Otherwise, no special syntax is
necessary. The {} bracket pair is used throughout TMQL already to wrap query
expressions. The only procurement are two predefined TMQL concepts (tmql:function and
tmql:return).
There is, though, an additional requirement for CTM to allow occurrence values to be wrapped
inside {} pairs. By itself this would imply that a CTM parser has to parse TMQL
expressions. To avoid this burden and to keep CTM parsers independent from TMQL expressions,
also the following can be allowed in CTM:
nr_employees isa tmql:function
tmql:return : {{
.... whatever, even with use of brackets {}
}}
The opening {{ (any number of brackets is possible as prolog) must only have one matching
set }}. This is simple and fast to implement.
For predicates this scheme runs similar:
is_NGO isa tmql:predicate
tmql:where : not $o <- part -> whole == // government
As the syntax of the boolean expression in the WHERE clause is quite restricted (one cannot
generate content with it), there would be no need for any terminators. But again, this
would imply that an (embedded) CTM parser would have to understand the syntax of boolean
expressions. To keep that agnostic, the same {} bracketing can be used; and as above, this
is can be kept optional if the end-of-line is terminating occurrence values:
is_NGO isa tmql:predicate
tmql:where : {{
not $o <- part -> whole = // government
}}
Structured Discussion
? should functions, predicates and templates be first-class topics?
- many people will find this very unconventional, are used to different syntax
- "un-conventional" is a marketing argument, not a technical argument
- "it is just a fu**ing syntax"
+ maybe, but the question is: use yet another new one or re-use an existing one?
+ it is a TMish view on existing concepts
+ directly reflects a TM-view on functions, no further explanation to a user necessary
+ also predefined functions and predicates can be presented that way
- it can also be done with conventional syntax,
- yes, but then more explanation is necessary how particular syntactic elements
have to be mapped onto a TMish view of them
- yes, but we need syntax for: functions, predicates and 'prefixes'
+ reduces syntax for TMQL
- yes, but it puts some burden onto CTM
- which is minimal ({{...}}...)
- dedicated syntax has less clutter
- every dedicated syntax is _expected_ to have less clutter
- and a second step to expose functions as topics has to be defined
- parsing gets complicated when functions are subclassed (my_func iko tmql:function)
- not really, the functionality must be in a TMQL processor already
- parsing gets more complicated when scoping is used
- maybe, not sure how much of an issue this is
- with dedicated syntax the requirements on the (function and predicate) body
can be tested better by a parser
+ it is not possible to 'hide' a function using subtypes of tmql:function
+ it is not possible to abuse scope
! Oslo 2007: this question will be refactored into a TMOL language
=> see 'refactorisation' issue
=> how functions and predicates can be declared is one thing
=> how TMQL 'experiences' these functions and predicates is another thing
