<div dir="ltr">I think you can linearize the scan of all the translation units by assuming all of the unknown modules will be in $PROJECT_MODULE_DIR/module_name.bmi, and when you find the interface unit that produces module_name, write out the build dependency there. This works less well for external dependencies. There's also the additional complication that the imported module name may not exist yet. That should be a build error, but shouldn't necessarily be a hard error during scanning and collating. Building unrelated source should still be possible, the keep-going option. <br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 11, 2019 at 3:30 PM Matthew Woehlke <<a href="mailto:mwoehlke.floss@gmail.com">mwoehlke.floss@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 11/02/2019 14.05, Ben Boeckel wrote:<br>
> On Mon, Feb 11, 2019 at 11:56:23 -0500, Matthew Woehlke wrote:<br>
>> I think there might be some communication confusion here. "Determining<br>
>> dependencies" can mean two things: determining the *identity* of<br>
>> dependencies, or determining the *location* of dependencies. Determining<br>
>> the *identity* is, indeed, as easy for modules as for includes. It's the<br>
>> *location* that's a problem. For includes, the compiler already has a<br>
>> set of include paths which it can scan (relatively quick; this is just<br>
>> directory listing) to determine location.<br>
>><br>
>> For modules... I'm not sure how this is going to work. This actually<br>
>> relates to the open question of how we "ship" modules. But ignore that<br>
>> for a second.<br>
> <br>
> In CMake, the identity is output by the `scan` step. `collate` puts this<br>
> together with output paths to create full paths to locations of modules.<br>
<br>
For project-internal stuff, yes. What about externals? Do those need to<br>
be scanned also? If so, how?<br>
<br>
I think/hope the answer is going to be "no", but while I think we<br>
generally have consensus how we would *like* that to work, I don't know<br>
that we have anything that *does* work yet.<br>
<br>
I'm not saying that needs to be a "blocker", just that it should be kept<br>
in mind.<br>
<br>
>> Compared to includes, we have more trouble with finding the location of<br>
>> modules provided by the project. To do that, we have to scan every<br>
>> source file in the project (or at least, in every library of the project<br>
>> that the current TU uses, including the TU's own component). This most<br>
>> assuredly *is* slower. (Unless we use module maps...)<br>
> <br>
> The `collate` step handles modules generated by other source files and<br>
> outputting the information required by the build tool to get the<br>
> dependency graph correct.<br>
<br>
Right. That's the "scan[ning] every source file in [...] every library<br>
of the project that the current TU uses". The good news is we don't need<br>
to do that *per TU* (we can do it "once"¹ and reuse the result), but as<br>
you originally noted, it's a non-trivial difference compared to what we<br>
do now.<br>
<br>
Mostly I was picking at the claim that "It is no slower to determine the<br>
dependencies of a modular C++ file than a normal C++ file today".<br>
Strictly speaking, this is true iff you want the dependency *identities*<br>
but don't care about their *locations*. For a single TU, getting the<br>
*locations* is decidedly slower. In the context of a complete build,<br>
however, it's harder to say, since the extra work can be done in<br>
parallel and the cost is effectively spread out across multiple TU's.<br>
<br>
(¹ Modulo need to recompute when things change, but at worst, once per<br>
build.)<br>
<br>
>> What about modules *external* to the project? If they are shipped<br>
>> already built, then presumably things aren't much different than<br>
>> includes; we expect them to just be there already.<br>
> <br>
> I think there is consensus that compiled modules do *not* get shipped.<br>
> Some other format (possibly IPR[1]) would be provided.<br>
<br>
Right... at which point we have a caching problem, but (despite being<br>
one of the two hard problems in programming) I think we can deal with<br>
that :-).<br>
<br>
Hmm... but, is it then the worst case that the build tool will have to<br>
find the module interface files for all modules that are needed but do<br>
*not* come from local sources, and generate build rules for those? (But<br>
at least that's back to being a directory-search operation and not a<br>
file-parse operation. At least I *assume*/hope that's the intent of IPR<br>
or whatever!) Or do we expect this to be the compiler's job?<br>
<br>
(In any case, we're straying from IS territory, though it's probably<br>
still useful to have an idea of the answers for such questions...)<br>
<br>
>> Where I could see this getting *really* ugly is if a) we have no module<br>
>> maps, and b) we have no solution for shipping already-built modules. If,<br>
>> in order to use the module from some external library, I first need to<br>
>> *build* that module for my own project... how do I do that? Scan every<br>
>> source file *on the entire system*? That's not just slower, it's not<br>
>> even feasible. Fortunately, I think we all want to not have to go there...<br>
> <br>
> A module is what you need to consume the interface. You don't need the<br>
> source itself for that. The IPR would just say what symbols are provided<br>
> via that module (analogous to headers saying what symbols are available)<br>
> and the linker would do the same job it has always done and hook them up<br>
> when linking the providing library.<br>
<br>
Right... the point was more that we *need* such a thing :-).<br>
<br>
More specifically, we need *something* which is a) portable, and b)<br>
ensures that we can get BMI's from external packages. The ways I see to<br>
do that are:<br>
<br>
- Per (my) above, ship sources and *scan* sources to figure out how to<br>
build modules. I think we all *really don't* want this.<br>
<br>
- Ship module maps to cut out the scanning step of the above. These<br>
would be *build* artifacts, i.e. build/packaging produces them.<br>
<br>
- Per (your) above, ship something that has a 1:1 mapping to modules<br>
that is a portable representation from which BMI's can be generated. (Or<br>
which the compiler / tooling can consume directly. Some tooling<br>
applications may be able to use these directly and never need a BMI.) I<br>
think most (all?) of us prefer this.<br>
<br>
For the last, we *may* still want/need some sort of "module map" to map<br>
names to module representation artifacts, but as per the middle option,<br>
this would be generated, not user-maintained.<br>
<br>
-- <br>
Matthew<br>
_______________________________________________<br>
Tooling mailing list<br>
<a href="mailto:Tooling@isocpp.open-std.org" target="_blank">Tooling@isocpp.open-std.org</a><br>
<a href="http://www.open-std.org/mailman/listinfo/tooling" rel="noreferrer" target="_blank">http://www.open-std.org/mailman/listinfo/tooling</a><br>
</blockquote></div>