From owner-sc22wg5+sc22wg5-dom9=www.open-std.org@open-std.org Mon Sep 25 01:55:37 2023 Return-Path: X-Original-To: sc22wg5-dom9 Delivered-To: sc22wg5-dom9@www.open-std.org Received: by www.open-std.org (Postfix, from userid 521) id 9F2F53569FB; Mon, 25 Sep 2023 01:55:37 +0200 (CEST) Delivered-To: sc22wg5@open-std.org Received: from nag-j.co.jp (bvdeuz19.secure.ne.jp [180.222.80.19]) by www.open-std.org (Postfix) with SMTP id 2D4CD3569C7 for ; Mon, 25 Sep 2023 01:55:31 +0200 (CEST) Received: (qmail 9816 invoked from network); 25 Sep 2023 08:55:28 +0900 Received: from unknown (HELO Maru10) (218.42.159.105) by 0 with SMTP; 25 Sep 2023 08:55:28 +0900 From: "Malcolm Cohen" To: "'WG5'" References: <20230912161937.ED0F6356986@www.open-std.org> <493F5A4F-8746-4CA4-ACF7-554B71168255@nasa.gov> <20230912184902.A227F35699F@www.open-std.org> <20230914051137.8F0F93569EF@www.open-std.org> <20230914132511.A9155356742@www.open-std.org> <20230919074924.5A64A356969@www.open-std.org> <20230919085702.1ECCF3569C0@www.open-std.org> <73963378-8AB0-447B-8BA0-EC90F07B0FEF@nasa.gov> <20230920085704.D845D3569F4@www.open-std.org> <20230920093749.0462C3569FD@www.open-std.org> In-Reply-To: Subject: RE: Do Concurrent with unspecified locality, calling pure procedures Date: Mon, 25 Sep 2023 08:55:28 +0900 Message-ID: <047201d9ef42$986b5030$c941f090$@nag-j.co.jp> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0473_01D9EF8E.085A2420" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQG/qXZQcGIQLpZ1F3Ca3tHLE/EEbAFGgQaEAuD8HXAB7pdlIQFRoAeAAhuw4kIC1WjZxwJ3QBx3AbrjyQ0B8n/94QMXn0msAfWBGYgClWAmvK+OLJRA Content-Language: ja Sender: owner-sc22wg5@open-std.org Precedence: bulk This is a multipart message in MIME format. ------=_NextPart_000_0473_01D9EF8E.085A2420 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi Jeff, Nitpick: Code is invalid. You want to make "B" INTENT(INOUT), otherwise the whole of B becomes undefined (at the call), and thus the whole of B is undefined on exit from the loop, which is probably not what you intend. After that fix, serial execution works fine. > I cannot see any way for the compiler to implement what we require there, which is that elements of T be localized We do not require that, because that is NOT one of the possible effects of unspecified locality - the standard never requires that unspecified locality behave as LOCAL or LOCAL_INIT. All elements that are written by multiple iterations become *undefined*. It is invalid to refer to the value of an undefined variable, whereas LOCAL/LOCAL_INIT leave the variable with its previous definition status and value. Now, a compiler auto-parallelising DO CONCURRENT may well implement some cases of unspecified locality by creating temps, just as if the variable were LOCAL. But the user program has no valid way to tell whether that happened, because the variable is undefined. > I cannot see any way for the compiler to implement Implementation by serial execution still works fine. > I'd just like to add the commonsense thing that we all agree on, which is that unspecified locality is a terrible idea, Such a note is not appropriate for the standard. > it seems catastrophic Lack of parallel execution (but possible retention of other optimisations) is not what I would call catastrophic. Catastrophic would be "runs noticeably slower than normal DO", or "inevitably gives the wrong answer". Here it will be approximately the same as normal DO (and may be faster if one of the other optimisations kicks in), just no auto-parallelisation. > in the case of procedure calls Well, if you're going to pass whole arrays around when you only want to read/write one or two elements, sure. That falls into "hiding information from the compiler", which is never a good idea when you are hoping for some kind of auto-parallelism. Or other optimisations for that matter. Cheers, -- ..............Malcolm Cohen, NAG Oxford/Tokyo. From: Jeff Hammond Sent: Wednesday, September 20, 2023 7:29 PM To: General J3 interest list Cc: WG5 ; Malcolm Cohen Subject: Do Concurrent with unspecified locality, calling pure procedures Can we add something about the fact that unspecified locality is nigh impossible when the code that is executed during the iteration is invisible to the compiler due to a procedure call? If I take the Peter example and put the loop body in a pure procedure, I cannot see any way for the compiler to implement what we require there, which is that elements of T be localized whenever K=L (or the entire array is localized). The alternative seems to be that procedure calls with arrays will disable parallelism unless the compile does whole-program analysis, possibly at link-time, to figure out what is defined or undefined by the procedure. I don't expect us to solve this. I'd just like to add the commonsense thing that we all agree on, which is that unspecified locality is a terrible idea, and that it seems catastrophic in the case of procedure calls. module w interface pure subroutine bar(J,K,L,T,A,B) implicit none integer, intent(in) :: J, K, L real, intent(in) :: A(:) real, intent(out) :: B(:) real, intent(inout) :: T(:) ! the implementation, found elsewhere, does this: ! T(K) = A(J) ! B(J) = T(L) end subroutine bar end interface end module w subroutine foo(N, A, B, T, K, L) use w implicit none integer, intent(in) :: N, K(N), L(N) real, intent(in) :: A(N) real, intent(out) :: B(N) real, intent(inout) :: T(N) do concurrent (integer :: J=1:N) call bar(J,K(J),L(J),T,A,B) end do end subroutine foo From: J3 > on behalf of Jeff Hammond via J3 > Date: Wednesday, 20. September 2023 at 12.37 To: General J3 interest list > Cc: Jeff Hammond >, WG5 >, Malcolm Cohen > Subject: [J3] [SC22WG5.6491] RE: [EXTERNAL] [BULK] Subobject of a variable External email: Use caution opening links or attachments Thanks, I have been slowly coming around to a similar conclusion, which is that Do Concurrent means "in any order" plus "user hints that auto-parallelization is desirable", which is to say, Do Concurrent may only be mapped to "!$omp parallel do" conditionally, after the compiler has proven that the loop is legal to auto-parallelize. In this interpretation, Concurrent is still quite powerful, because it gives the user a way to strongly encourage auto-parallelization, when compilers may not otherwise do so aggressively because auto-parallelization all Do loops that are legal to do so with threads is likely to hurt performance, because thread overhead is nontrivial. This should lead to high-quality implementations of auto-parallelization (as I know Damian and friends want) and sequential behavior only in pathological cases that most people aren't writing. It may interest some to know that yesterday during the OpenMP meeting, we decided to settle the problem of "!$omp loop" on Do Concurrent by adding a restriction that users must specify the locality of variables that will be written to in more than one iteration. This means that, in the context of OpenMP, Peter's problematic example becomes erroneous unless the programmer adds shared() appropriately, as you and Damian both suggest is the resolution to this. Anyways, our implementation likely requires very little change to reflect this, since Do Concurrent can be implemented with OpenACC "kernels", which causes the compiler to do auto-parallelization, based on what it can prove is possible. I imagine that Intel will have to do more work, to disable parallelism when it's not legal. Jeff On 20. Sep 2023, at 11.56, Malcolm Cohen via J3 > wrote: External email: Use caution opening links or attachments Hi Jeff, The other side of a variable being "undefined" is "having a specific value". No-one would expect that having a specific value propagates to subobjects (the rank or type would be different). > It is hard for me to understand how I can apply the locality inference rules when the inferred locality propagates, but the rules used for that inference don't. Rules don't "propagate". Rules are rules. I do not see anything odd about "If I define the whole of array X in an iteration, no other iteration can define the whole of array X" and "If I define an element of the array X in an iteration, no other iteration can define that element of X". Perhaps it might help to think that DO CONCURRENT is not in fact a variation of OpenMP !$OMP DO. That is not how it came about. DO CONCURRENT is two things: 1. A reworking of FORALL to remove the plethora of array temps that FORALL often mandates. 2. A standardised form of the directive "!DEC$IVDEP" (I probably have that spelled wrong - the manuals are in the office.) This directive declares that the loop iterations are independent, and that enables various useful loop optimisations, like software pipelining and so on. Now, in the "simple" cases (and there are a lot of simple cases), it can be easy to parallelise a DO CONCURRENT as originally designed (F2008), viz with everything "unspecified locality", but although parallelisability was a strong consideration, it was not the driving force behind the original design. The drawbacks of unspecified locality re parallelisation are what lead to the introduction of specified locality in F2018. Seriously, users wanting good performance should not be using unspecified locality, except perhaps in the simplest of loops. Actually, with my Software Engineering hat on, I'd say users should not be using unspecified locality ever, they should tell the compiler *and subsequent readers/maintainers of the code* what they want explicitly. And for the compiler, if it cannot work out what the locality of some variable is, it should just execute the loop serially (in any order). That will always give the correct answer, because loop iterations are independent (well, the user claimed they were, anyway), and do not depend of what the locality actually is. (If a "parallelise DO CONCURRENT" option is active, it might be a good idea to produce a warning when this happens.) Anecdote: When I explained DO CONCURRENT to some OpenMP programmers, they were aghast. "Do not ever execute it in parallel!" was their response, with the explanation "If we want a loop executed in parallel we will use !$OMP PARALLEL DO, if the compiler might create threads on any other loop, that would be excessive threads and would slow down the overall execution due to thread contention for resources." Cheers, -- ..............Malcolm Cohen, NAG Oxford/Tokyo. From: J3 > On Behalf Of Jeff Hammond via J3 Sent: Tuesday, September 19, 2023 10:59 PM To: Clune, Thomas L. (GSFC-6101) >; General J3 interest list > Cc: Jeff Hammond >; WG5 >; Malcolm Cohen > Subject: [J3] [SC22WG5.6488] [EXTERNAL] [BULK] Subobject of a variable Okay, but we are saying that, in the context of do concurrent, locality obviously propagates to subobjects, but (un)definedness obviously doesn't. This bothers me because we use (un)definedness to infer the locality of variables with unspecified locality. It is hard for me to understand how I can apply the locality inference rules when the inferred locality propagates, but the rules used for that inference don't. Jeff From: Clune, Thomas L. (GSFC-6101) < thomas.l.clune@nasa.gov> Date: Tuesday, 19. September 2023 at 16.17 To: General J3 interest list < j3@mailman.j3-fortran.org> Cc: Jeff Hammond < jehammond@nvidia.com>, WG5 < sc22wg5@open-std.org>, Malcolm Cohen < malcolm@nag-j.co.jp> Subject: Re: [EXTERNAL] [BULK] [J3] [SC22WG5.6486] Subobject of a variable External email: Use caution opening links or attachments I do not find this situation odd. Lots of things don't propagate to subobjects. E.g., if an array is allocatable, it's elements are not. If an array is pointer, it's elements are not. And "undefinedness" is very much a property of subobjects. Indeed, a variable is only defined if all of its subobjects are defined. (I hope I have that right.) * Tom From: J3 < j3-bounces@mailman.j3-fortran.org> on behalf of j3 < j3@mailman.j3-fortran.org> Reply-To: j3 < j3@mailman.j3-fortran.org> Date: Tuesday, September 19, 2023 at 4:57 AM To: j3 < j3@mailman.j3-fortran.org> Cc: Jeff Hammond < jehammond@nvidia.com>, WG5 < sc22wg5@open-std.org>, Malcolm Cohen < malcolm@nag-j.co.jp> Subject: [EXTERNAL] [BULK] [J3] [SC22WG5.6486] Subobject of a variable CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. If a variable has unspecified locality, - if it is referenced in an iteration it shall either be previously defined during that iteration, or shall not be defined or become undefined during any other iteration; if it is defined or becomes undefined by more than one iteration it becomes undefined when the loop terminates; real, dimension(10) :: A do concurrent (integer :: k=1:size(A)) A(k) = k if (k.eq.1) A(10) = 0 end do As best I can tell, A(1:9) are defined after the loop terminates, while A(10) and A are not. Does the fact that A is undefined before and becomes undefined during this loop imply anything about the undefined-ness of its elements during the loop or must one reason about them strictly independently, based on the element-wise defined/undefined-ness? Now suppose I make A shared. Malcolm said all the elements of A are shared as well "It's not written there now, because, like, how could it possibly be anything else?" real, dimension(10) :: A do concurrent (integer :: k=1:size(A)) shared(A) A(k) = k if (k.eq.1) A(10) = 0 end do It is odd to me that the shared-ness of A propagates to its elements but the undefined-ness does not. Why must I reason about the element-wise behavior of an array independent of the array itself when worrying about whether a variable becomes undefined or defined during a loop, but in the same context, they are obviously not independent regarding locality? Jeff On 19. Sep 2023, at 10.49, Malcolm Cohen via J3 < j3@mailman.j3-fortran.org> wrote: External email: Use caution opening links or attachments Hi Brad, There is no doubt that subobject includes array elements. The point is that a subobject of a variable is itself a variable. So nothing more needs to be said on that front. But in the case of a subobject of a constant, that is often not constant. E.g. INTEGER,PARAMETER :: x(10) = [ 1,2,3,4,5,6,7,8,9,10 ] READ *,I PRINT *,x(I) Here, X(I) is not a variable, and is also not a constant. It is a non-constant subobject of a constant! That is what those words are there for. They are not intended to imply that subobjects of a variable are not data objects. They are simply because they are also variables, indeed, portions that can be defined and redefined separately, and being capable of being defined and redefined makes them variables. We could add a NOTE to say that "A subobject of a variable is a variable, but a subobject of a constant is not necessarily itself constant, e.g. if it has a non-constant subscript." Not sure it is really worth it though, at least, not until we decide what to do with our overly-large set of terms. Cheers, -- ..............Malcolm Cohen, NAG Oxford/Tokyo. From: Brad Richardson < everythingfunctional@protonmail.com> Sent: Thursday, September 14, 2023 10:25 PM To: General J3 interest list < j3@mailman.j3-fortran.org>; 'WG5' < sc22wg5@open-std.org> Subject: [ukfortran] [SC22WG5.6484] [J3] RE: [EXTERNAL] [BULK] are race conditions allowed in do concurrent? Hi Malcolm, > >variable/data entity/data object > >[are] circular, and doesn't seem to include individual array elements > Yes it does, see 3.138 subobject. While I see that it probably intends to include it, what it actually says is subobject of a constant so the edit would be to change it to subobject of a variable or constant I don't think the circularity is causing too much confusion, so I don't see much urgency in trying to fix it, and don't quite see an easy way to do so. I think we can leave it for now. Regards, Brad On Thu, 2023-09-14 at 14:04 +0900, Malcolm Cohen via J3 wrote: Hi Brad, There is no contradiction, subtle or otherwise, here. Locality-specs are only for named variables. However, as noted, subobjects of variables are also variables. > The way I've heard "definition" applied to arrays is that definition of an element of an array does not constitute definition of the variable, Correct (except in the trivial case of an array with a single element). This is all spelled out explicitly: defining every "part" of a multi-part variable defines the variable. I can see some potential wording improvements for the next revision, but nothing that rises to the level of a defect. E.g. it may be better to mention that "if a variable has SHARED locality, all of its subobjects have SHARED locality". It's not written there now, because, like, how could it possibly be anything else? There is nothing else it could be. But making it explicit would lead naturally into writing "If a variable with SHARED locality is defined." instead of the current "If it is defined.". >variable/data entity/data object >[are] circular, and doesn't seem to include individual array elements Yes it does, see 3.138 subobject. The circularity is unfortunate, but they are all reasonably worded for comprehensibility (not for trail-following). So it's either deleting one of them (ugh) or rewording one to break the circle without making it hard to understand (it's not obvious exactly what to do here, which is why it is still like this). If it were a glossary of terms, instead of a list of definitions, there would be no problem with circularity, right? Anyway, I agree it would be good to fix it as long as it doesn't make any individual term hard to understand. Given our experience this time around, I would be in favour of pruning our terms and definitions, down to a more minimal set. If there are "useful" terms that do not lend themselves to trivial drop-in replacement, and I think there are, we could consider adding a "Glossary" (of "useful terms", but probably better not to call them "terms"!). Cheers, -- ..............Malcolm Cohen, NAG Oxford/Tokyo. From: Brad Richardson < everythingfunctional@protonmail.com> Sent: Wednesday, September 13, 2023 3:49 AM To: General J3 interest list < j3@mailman.j3-fortran.org>; WG5 < sc22wg5@open-std.org> Subject: [ukfortran] [SC22WG5.6480] [EXTERNAL] [BULK] [J3] are race conditions allowed in do concurrent? Hmm... I decided to work through this a bit more thoroughly. From the syntax definitions: R902 variable is designator or function-reference R901 designator is object-name or array-element or array-section or coindexed-named-object or complex-part-designator or structure-component or substring Ok, so according to the syntax an array element is a variable. From the terms and definitions: variable data entity (3.41) that can be defined (3.48) and redefined during execution of a program data entity data object (3.42), result of the evaluation of an expression, or the result of the execution of a function reference data object constant, variable, or subobject of a constant which is circular, and doesn't seem to include individual array elements. So now the question is which "definition" does the DO CONCURRENT constraint refer to? From the syntax definitions: R1129 concurrent-locality is [ locality-spec ]... R1130 locality-spec is LOCAL ( variable-name-list ) or LOCAL_INIT ( variable-name-list ) or REDUCE ( reduce-operation : variable-name-list ) or SHARED ( variable-name-list ) or DEFAULT ( NONE ) only allows variable-name, not designator, so it's not exactly clear. The "constraint" appears in normative text, I.e. If a variable has which is not hyperlinked and is not in a font indicating reference to the syntax term. My reading of this is that we have a subtle contradiction between the syntax and normative terms. My guess is that the syntax meaning (i.e. an array element is a variable) is what is intended for the rules here, and thus it's valid to (un)define *different* elements of an array in separate iterations, but not valid to (un)define and reference the same element of an array in separate iterations. I think it would be worth "fixing" the terms and definitions to eliminate the circular definition and make it consistent with the syntax. Regards, Brad On Tue, 2023-09-12 at 17:37 +0000, Clune, Thomas L. (GSFC-6101) wrote: "Variable" does not mean what we think it means: <> <> And includes . But I for one would not mind an explanatory note that clarifies the common use case of array elements in this context. From:J3 < j3-bounces@mailman.j3-fortran.org> on behalf of j3 < j3@mailman.j3-fortran.org> Reply-To: j3 < j3@mailman.j3-fortran.org> Date: Tuesday, September 12, 2023 at 12:19 PM To: j3 < j3@mailman.j3-fortran.org>, WG5 < sc22wg5@open-std.org> Cc: Brad Richardson < everythingfunctional@protonmail.com> Subject: [EXTERNAL] [BULK] [J3] [SC22WG5.6477] are race conditions allowed in do concurrent? CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. Hi Jeff, Based on " If it is defined or becomes undefined during any iteration, it shall not be referenced, defined, or become undefined during any other iteration.", the provide program is invalid, because `x` becomes defined in more than one iteration. The way I've heard "definition" applied to arrays is that definition of an element of an array does not constitute definition of the variable, because it is then only partially defined. I.e. an array is only defined once each of its elements has been defined. My opinion is that the above constraint simply does not adequately describe/is still ambiguous about the case of arrays. I think an interp would be warranted, and we should say that an element of an array should have the same constraint as above if the variable is shared. Regards, Brad On Tue, 2023-09-12 at 14:43 +0000, Jeff Hammond via J3 wrote: Is this a legal program in Fortran with defined behavior? If it is illegal or the behavior is undefined, where do we say that? program main integer :: k real :: x x = 0.0 do concurrent (k=1:2) shared(x) x = real(k) end do print*,x end program main Is that what the following is intended to do? "If a variable has SHARED locality, appearances of the variable within the DO CONCURRENT construct refer to the variable in the innermost executable construct or scoping unit that includes the DO CONCURRENT construct. If it is defined or becomes undefined during any iteration, it shall not be referenced, defined, or become undefined during any other iteration." If so, then I am struggling to apply this to arrays. When we say a variable has shared locality, is the variable the array or the element? We can only put array names in SHARED(), but applying the text regarding "to become defined" from the perspective of arrays appears to have interesting consequences. Thanks, Jeff ------=_NextPart_000_0473_01D9EF8E.085A2420 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi Jeff,

 

Nitpick: Code is = invalid. You want to make “B” INTENT(INOUT), otherwise the = whole of B becomes undefined (at the call), and thus the whole of B is = undefined on exit from the loop, which is probably not what you = intend.

 

After that fix, = serial execution works fine.

 

> I cannot see any way for = the compiler to implement what we require there, which is that elements = of T be localized

 

We do not require = that, because that is NOT one of the possible effects of unspecified = locality – the standard never requires that unspecified locality = behave as LOCAL or LOCAL_INIT. All elements that are written by multiple = iterations become *undefined*. It is invalid to refer to the = value of an undefined variable, whereas LOCAL/LOCAL_INIT leave the = variable with its previous definition status and = value.

 

Now, a compiler = auto-parallelising DO CONCURRENT may well implement some cases of = unspecified locality by creating temps, just as if the variable were = LOCAL. But the user program has no valid way to tell whether that = happened, because the variable is undefined.

 

> I cannot see any way for = the compiler to implement

 

Implementation by serial execution still = works fine.

 

> I’d just like to = add the commonsense thing that we all agree on, which is that = unspecified locality is a terrible idea,

 

Such a = note is not appropriate for the standard.

 

> it = seems catastrophic

 

Lack of = parallel execution (but possible retention of other optimisations) is = not what I would call catastrophic. Catastrophic would be “runs = noticeably slower than normal DO”, or “inevitably gives the = wrong answer”. Here it will be approximately the same as normal DO = (and may be faster if one of the other optimisations kicks in), just no = auto-parallelisation.

 

> in = the case of procedure calls

 

Well, if = you’re going to pass whole arrays around when you only want to = read/write one or two elements, sure. That falls into “hiding = information from the compiler”, which is never a good idea when = you are hoping for some kind of auto-parallelism. Or other optimisations = for that matter.

 

Cheers,

--

..............Malcolm Cohen, NAG = Oxford/Tokyo.

 

From: Jeff Hammond <jehammond@nvidia.com> =
Sent: Wednesday, September 20, 2023 7:29 PM
To: = General J3 interest list <j3@mailman.j3-fortran.org>
Cc: = WG5 <sc22wg5@open-std.org>; Malcolm Cohen = <malcolm@nag-j.co.jp>
Subject: Do Concurrent with = unspecified locality, calling pure = procedures

 

Can we add something about the = fact that unspecified locality is nigh impossible when the code that is = executed during the iteration is invisible to the compiler due to a = procedure call?

 

If I = take the Peter example and put the loop body in a pure procedure, I = cannot see any way for the compiler to implement what we require there, = which is that elements of T be localized whenever K=3DL (or the entire = array is localized).


The alternative seems to be = that procedure calls with arrays will disable parallelism unless the = compile does whole-program analysis, possibly at link-time, to figure = out what is defined or undefined by the = procedure.

 

I = don’t expect us to solve this.  I’d just like to add = the commonsense thing that we all agree on, which is that unspecified = locality is a terrible idea, and that it seems catastrophic in the case = of procedure calls.

 

module = w

interface

pure subroutine = bar(J,K,L,T,A,B)

  implicit none

  integer, = intent(in) :: J, K, L

  real, intent(in) :: = A(:)

  real, intent(out) :: = B(:)

  real, intent(inout) :: = T(:)

  ! the implementation, found = elsewhere, = does this:

  ! T(K) =3D = A(J)

  ! B(J) =3D = T(L)

end subroutine bar

end = interface

end module w

 

subroutine foo(N, A, = B, T, K, L)

  use w

  implicit = none

  integer, intent(in) :: N, K(N), = L(N)

  real, intent(in) :: = A(N)

  real, intent(out) :: = B(N)

  real, intent(inout) :: = T(N)

  do concurrent (integer :: = J=3D1:N)

    call = bar(J,K(J),L(J),T,A,B)

  end do

end subroutine = foo

 

From: J3 <j3-bounces@mailman.j3-f= ortran.org> on behalf of Jeff Hammond via J3 <j3@mailman.j3-fortran.org&g= t;
Date: Wednesday, 20. September 2023 at 12.37
To: = General J3 interest list <j3@mailman.j3-fortran.org&g= t;
Cc: Jeff Hammond <jehammond@nvidia.com>, WG5 = <sc22wg5@open-std.org>, = Malcolm Cohen <malcolm@nag-j.co.jp>
Sub= ject: [J3] [SC22WG5.6491] RE: [EXTERNAL] [BULK] Subobject of a = variable

Ex= ternal email: Use caution opening links or attachments =

 

Thanks, I have been = slowly coming around to a similar conclusion, which is that Do = Concurrent means “in any order” plus “user hints that = auto-parallelization is desirable”, which is to say, Do Concurrent = may only be mapped to “!$omp parallel do" conditionally, = after the compiler has proven that the loop is legal to = auto-parallelize.

 

In this = interpretation, Concurrent is still quite powerful, because it gives the = user a way to strongly encourage auto-parallelization, when compilers = may not otherwise do so aggressively because auto-parallelization all Do = loops that are legal to do so with threads is likely to hurt = performance, because thread overhead is nontrivial.  This should = lead to high-quality implementations of auto-parallelization (as I know = Damian and friends want) and sequential behavior only in pathological = cases that most people aren’t writing. =

 

It may interest some = to know that yesterday during the OpenMP meeting, we decided to settle = the problem of “!$omp loop” on Do Concurrent by adding a = restriction that users must specify the locality of variables that will = be written to in more than one iteration.  This means that, in the = context of OpenMP, Peter’s problematic example becomes erroneous = unless the programmer adds shared() appropriately, as you and Damian = both suggest is the resolution to = this.

 

Anyways, our = implementation likely requires very little change to reflect this, since = Do Concurrent can be implemented with OpenACC “kernels”, = which causes the compiler to do auto-parallelization, based on what it = can prove is possible.  I imagine that Intel will have to do more = work, to disable parallelism when it’s not = legal.

 

Jeff

 

 

On 20. Sep 2023, at = 11.56, Malcolm Cohen via J3 <j3@mailman.j3-fortran.org&g= t; wrote:

 

Ex= ternal email: Use caution opening links or attachments=

 

Hi = Jeff,

 

The other side of a = variable being “undefined” is “having a specific = value”.

 

No-one would expect = that having a specific value propagates to subobjects (the rank or type = would be different).

 

> It is hard for me to understand how I can = apply the locality inference rules when the inferred locality = propagates, but the rules used for that inference = don’t.

 

Rules = don’t “propagate”. Rules are = rules.

 

I do not = see anything odd about “If I define the whole of array X in an = iteration, no other iteration can define the whole of array X” and = “If I define an element of the array X in an iteration, no other = iteration can define that element of = X”.

 

Perhaps = it might help to think that DO CONCURRENT is not in fact a variation of = OpenMP !$OMP DO. That is not how it came about. DO CONCURRENT is two = things:

  1. A = reworking of FORALL to remove the plethora of array temps that FORALL = often mandates.
  2. A = standardised form of the directive “!DEC$IVDEP” (I probably = have that spelled wrong – the manuals are in the office.) This = directive declares that the loop iterations are independent, and that = enables various useful loop optimisations, like software pipelining and = so on.

 

Now, in the = “simple” cases (and there are a lot of simple cases), it can = be easy to parallelise a DO CONCURRENT as originally designed (F2008), = viz with everything “unspecified locality”, but although = parallelisability was a strong consideration, it was not the driving = force behind the original design.

 

The drawbacks of = unspecified locality re parallelisation are what lead to the = introduction of specified locality in = F2018.

 

Seriously, users = wanting good performance should not be using unspecified locality, = except perhaps in the simplest of loops. Actually, with my Software = Engineering hat on, I’d say users should not be using unspecified = locality ever, they should tell the compiler *and subsequent = readers/maintainers of the code* what they want = explicitly.

 

And for the compiler, = if it cannot work out what the locality of some variable is, it should = just execute the loop serially (in any order). That will always give the = correct answer, because loop iterations are independent (well, the user = claimed they were, anyway), and do not depend of what the locality = actually is. (If a “parallelise DO CONCURRENT” option is = active, it might be a good idea to produce a warning when this = happens.)

 

Anecdote: When I = explained DO CONCURRENT to some OpenMP programmers, they were aghast. = “Do not ever execute it in parallel!” was their response, = with the explanation “If we want a loop executed in parallel we = will use !$OMP PARALLEL DO, if the compiler might create threads on any = other loop, that would be excessive threads and would slow down the = overall execution due to thread contention for = resources.”

 

Cheers,

--

..............Malcolm Cohen, NAG = Oxford/Tokyo.

 

From: J3 <j3-bounces@mailman.j3-f= ortran.org> On = Behalf Of Jeff = Hammond via J3
Sent: Tuesday, September 19, 2023 = 10:59 PM
To: Clune, Thomas L. (GSFC-6101) = <thomas.l.clune@nasa.gov>; = General J3 interest list <j3@mailman.j3-fortran.org&g= t;
Cc: Jeff = Hammond <jehammond@nvidia.com>; WG5 = <sc22wg5@open-std.org>; = Malcolm Cohen <malcolm@nag-j.co.jp>
Sub= ject: [J3] = [SC22WG5.6488] [EXTERNAL] [BULK] Subobject of a = variable

 

Okay, = but we are saying that, in the context of do concurrent, locality = obviously propagates to subobjects, but (un)definedness obviously = doesn’t.

 

This = bothers me because we use (un)definedness to infer the locality of = variables with unspecified locality.  It is hard for me to = understand how I can apply the locality inference rules when the = inferred locality propagates, but the rules used for that inference = don’t.

 

Jeff

 

From: Clune, Thomas L. (GSFC-6101) <thomas.l.clune@nasa.gov>
Date: Tuesday, 19. September = 2023 at 16.17
To: General J3 interest list = <j3@mailman.j3-fortran.org>
Cc: Jeff Hammond = <jehammond@nvidia.com>, WG5 <sc22wg5@open-std.org>, Malcolm Cohen = <malcolm@nag-j.co.jp>
Subject: Re: [EXTERNAL] [BULK] = [J3] [SC22WG5.6486] Subobject of a = variable

Ex= ternal email: Use caution opening links or = attachments

 

I do = not find this situation odd.    Lots of things = don’t propagate to subobjects.      E.g., = if an array is allocatable, it’s elements are not.  If an = array is pointer, it’s elements are = not.

 

And = “undefinedness” is very much a property of = subobjects.   Indeed, a variable is only defined if all of its = subobjects are defined.  (I hope I have that = right.)

 

  • Tom

 

From: J3 <j3-bounces@mailman.j3-fortran.org> on behalf of j3 = <j3@mailman.j3-fortran.org>
Reply-To: j3 <j3@mailman.j3-fortran.org>
Date: Tuesday, September 19, = 2023 at 4:57 AM
To: j3 <j3@mailman.j3-fortran.org>
Cc: Jeff Hammond = <jehammond@nvidia.com>, WG5 <sc22wg5@open-std.org>, Malcolm Cohen = <malcolm@nag-j.co.jp>
Subject: [EXTERNAL] [BULK] [J3] = [SC22WG5.6486] Subobject of a = variable

 

CAUTION: This email originated from outside of NASA.  = Please take care when clicking links or opening attachments.  Use = the "Report Message" button to report suspicious messages to = the NASA SOC.



<= div>

If a variable has = unspecified locality,=

- if it is referenced = in an iteration it shall either be previously defined during that = iteration, or shall not be defined or become undefined during any other = iteration; if it is defined or becomes undefined by more than one = iteration it becomes undefined when the loop terminates;=

real, = dimension(10) :: A

 

<= div style=3D'margin-left:36.0pt'>

do concurrent (integer :: = k=3D1:size(A))

  A(k) =3D = k

  if (k.eq.1) A(10) =3D = 0

end = do

 

<= div style=3D'margin-left:36.0pt'>

As best I can tell, A(1:9) are defined after = the loop terminates, while A(10) and A are not.  Does the fact that = A is undefined before and becomes undefined during this loop imply = anything about the undefined-ness of its elements during the loop or = must one reason about them strictly independently, based on the = element-wise = defined/undefined-ness?

 

<= div style=3D'margin-left:36.0pt'>

Now suppose I make A shared.  Malcolm = said all the elements of A are shared as well "It’s not = written there now, because, like, how could it possibly be anything = else?”

 

<= blockquote = style=3D'margin-left:30.0pt;margin-top:5.0pt;margin-right:0cm;margin-bott= om:5.0pt'>

real, = dimension(10) :: A

 

<= div style=3D'margin-left:36.0pt'>

do concurrent (integer :: k=3D1:size(A)) = shared(A)

  A(k) =3D = k

  if (k.eq.1) A(10) =3D = 0

end = do

 

<= div style=3D'margin-left:36.0pt'>

It is odd to me that the shared-ness of A = propagates to its elements but the undefined-ness does not.  Why = must I reason about the element-wise behavior of an array independent of = the array itself when worrying about whether a variable becomes = undefined or defined during a loop, but in the same context, they are = obviously not independent regarding = locality?

 

<= div style=3D'margin-left:36.0pt'>

Jeff

 



On 19. Sep 2023, at 10.49, Malcolm Cohen via = J3 <j3@mailman.j3-fortran.org> = wrote:

 

Ex= ternal email: Use caution opening links or = attachments

 

Hi = Brad,

 

<= div style=3D'margin-left:36.0pt'>

There is no doubt that subobject includes = array elements.

 

<= div style=3D'margin-left:36.0pt'>

The point is that a subobject of a variable = is itself a variable. So nothing more needs to be said on that = front.

 

<= div style=3D'margin-left:36.0pt'>

But in the case of a subobject of a constant, = that is often not constant. = E.g.

   INTEGER,PARAMETER :: x(10) =3D [ = 1,2,3,4,5,6,7,8,9,10 ]

   READ = *,I

   PRINT *,x(I) 

 

<= div style=3D'margin-left:36.0pt'>

Here, X(I) is not a variable, and is also not = a constant. It is a non-constant subobject of a = constant!

 

<= div style=3D'margin-left:36.0pt'>

That is what those words are there for. They = are not intended to imply that subobjects of a variable are not data = objects. They are simply because they are also variables, indeed, = portions that can be defined and redefined separately, and being capable = of being defined and redefined makes them = variables.

 

<= div style=3D'margin-left:36.0pt'>

We could add a NOTE to say that “A = subobject of a variable is a variable, but a subobject of a constant is = not necessarily itself constant, e.g. if it has a non-constant = subscript.” Not sure it is really worth it though, at least, not = until we decide what to do with our overly-large set of = terms.

 

<= div style=3D'margin-left:36.0pt'>

Cheers,

=

--

..............Malcolm Cohen, NAG = Oxford/Tokyo.

 

<= div style=3D'border:none;border-top:solid windowtext 1.0pt;padding:3.0pt = 0cm 0cm 0cm;border-color:currentcolor;border-image: none'>

From: Brad Richardson <everythingfunctional@protonmail.= com> 
Sent: Thursday, September 14, 2023 = 10:25 PM
To: General J3 interest list = <j3@mailman.j3-fortran.org= >; 'WG5' = <sc22wg5@open-std.org<= span lang=3DEN-US = style=3D'font-size:11.0pt'>>
Subject: [ukfortran] [SC22WG5.6484] = [J3] RE: [EXTERNAL] [BULK] are race conditions allowed in do = concurrent?

 

<= div>

Hi = Malcolm,

 

=

> >variable/data = entity/data object

> >[are] circular, and doesn't seem to = include individual array = elements

 

=

> Yes it does, see 3.138 = subobject.

 

=

While I see that it probably = intends to include it, what it actually says = is

 

=

subobject of a = constant

 

=

so the edit would be to change = it to

 

=

subobject of a variable or = constant

 

=

I don't think the circularity is = causing too much confusion, so I don't see much urgency in trying to fix = it, and don't quite see an easy way to do so. I think we can leave it = for now.

 

=

Regards,

Brad

 

=

On Thu, 2023-09-14 at 14:04 = +0900, Malcolm Cohen via J3 = wrote:

Hi = Brad,

 

<= div style=3D'margin-left:36.0pt'>

There is no contradiction, subtle or = otherwise, here.

 

<= div style=3D'margin-left:36.0pt'>

Locality-specs are only for named variables. = However, as noted, subobjects of variables are also = variables.

 

<= div style=3D'margin-left:36.0pt'>

> The way I've heard = "definition" applied to arrays is that definition of an = element of an array does not constitute definition of the = variable,

 

<= div style=3D'margin-left:36.0pt'>

Correct (except in the trivial case of an = array with a single element). This is all spelled out explicitly: = defining every “part” of a multi-part variable defines the = variable.

 

<= div style=3D'margin-left:36.0pt'>

I can see some potential wording improvements = for the next revision, but nothing that rises to the level of a = defect.

 

<= div style=3D'margin-left:36.0pt'>

E.g. it may be better to mention that = “if a variable has SHARED locality, all of its subobjects have = SHARED locality”. It’s not written there now, because, like, = how could it possibly be anything else? There is nothing else it could = be.  But making it explicit would lead naturally into writing = “If a variable with SHARED locality is defined…” = instead of the current “If it is = defined…”.

 

<= div style=3D'margin-left:36.0pt'>

>variable/data entity/data = object

>[are] circular, and doesn't seem to = include individual array = elements

 

<= div style=3D'margin-left:36.0pt'>

Yes it does, see 3.138 = subobject.

 

<= div style=3D'margin-left:36.0pt'>

The circularity is unfortunate, but they are = all reasonably worded for comprehensibility (not for trail-following). = So it’s either deleting one of them (ugh) or rewording one to = break the circle without making it hard to understand (it’s not = obvious exactly what to do here, which is why it is still like = this).

 

<= div style=3D'margin-left:36.0pt'>

If it were a glossary of terms, instead of a = list of definitions, there would be no problem with circularity, = right?

 

<= div style=3D'margin-left:36.0pt'>

Anyway, I agree it would be good to fix it as = long as it doesn’t make any individual term hard to = understand.

 

<= div style=3D'margin-left:36.0pt'>

Given our experience this time around, I = would be in favour of pruning our terms and definitions, down to a more = minimal set. If there are “useful” terms that do not lend = themselves to trivial drop-in replacement, and I think there are, we = could consider adding a “Glossary” (of “useful = terms”, but probably better not to call them = “terms”!).

 

<= div style=3D'margin-left:36.0pt'>

Cheers,

=

--

..............Malcolm Cohen, NAG = Oxford/Tokyo.

 

<= div style=3D'border:none;border-top:solid windowtext 1.0pt;padding:3.0pt = 0cm 0cm 0cm;border-color:currentcolor;border-image: none'>

From: Brad Richardson <everythingfunctional@protonmail.= com> 
Sent: Wednesday, September 13, 2023 = 3:49 AM
To: General J3 interest list = <j3@mailman.j3-fortran.org= >; WG5 = <sc22wg5@open-std.org<= span lang=3DEN-US = style=3D'font-size:11.0pt'>>
Subject: [ukfortran] [SC22WG5.6480] = [EXTERNAL] [BULK] [J3] are race conditions allowed in do = concurrent?

 

<= div>

Hmm... I decided to work through = this a bit more = thoroughly.

 

=

From the syntax = definitions:

 

=

R902

variable

is = designator

or = function-reference

 

=

R901

designator

is = object-name

or = array-element

or = array-section

or = coindexed-named-object

<= div style=3D'margin-left:36.0pt'>

or = complex-part-designator

=

or = structure-component

or = substring

 

=

 

=

Ok, so according to the syntax = an array element is a = variable.

 

=

From the terms and = definitions:

 

=

variable

data entity (3.41) that can be = defined (3.48) and redefined during execution of a = program

 

=

data = entity

data object (3.42), result of the evaluation = of an expression, or the result of the execution of a function = reference

 

=

data = object

constant, variable, or subobject of a = constant

 

=

which is circular, and doesn't = seem to include individual array = elements.

 

=

So now the question is which = "definition" does the DO CONCURRENT constraint refer = to?

 

=

From the syntax = definitions:

 

=

R1129

<= div>

concurrent-locality

is

[ locality-spec = ]...

 

=

R1130

<= div>

locality-spec

is LOCAL = ( variable-name-list = )

or LOCAL_INIT ( variable-name-list = )

or REDUCE ( reduce-operation : = variable-name-list = )

or SHARED ( variable-name-list = )

or DEFAULT ( NONE = )

 

=

only allows variable-name, not = designator, so it's not exactly = clear.

 

=

The "constraint" = appears in normative text, = I.e.

 

=

If a variable = has

 

=

which is not hyperlinked and is = not in a font indicating reference to the syntax = term.

 

=

My reading of this is that we = have a subtle contradiction between the syntax and normative terms. My = guess is that the syntax meaning (i.e. an array element is a variable) = is what is intended for the rules here, and thus it's valid to = (un)define *different* elements of an array in separate iterations, but = not valid to (un)define and reference the same element of an array in = separate = iterations.

 

=

I think it would be worth = "fixing" the terms and definitions to eliminate the circular = definition and make it consistent with the = syntax.

 

=

Regards,

Brad

 

=

On Tue, 2023-09-12 at 17:37 = +0000, Clune, Thomas L. (GSFC-6101) = wrote:

“Variable” does not mean what we = think it means:     <variable>  =  <<is>>  <designator> <<or>> = <function-reference>

 

<= div style=3D'margin-left:36.0pt'>

And <designator> includes = <array-element>.

 

<= div style=3D'margin-left:36.0pt'>

But I for one would not mind an explanatory = note that clarifies the common use case of array elements in this = context. 

 

<= div style=3D'margin-left:36.0pt'>

 

<= div style=3D'margin-left:36.0pt'>

 

From:J3 <j3-bounces@mailman.j3-fortran.or= g> on behalf = of j3 <j3@mailman.j3-fortran.org= >
Reply-To: j3 <j3@mailman.j3-fortran.org= >
Date: Tuesday, September 12, = 2023 at 12:19 PM
To: j3 <j3@mailman.j3-fortran.org= >, WG5 = <sc22wg5@open-std.org<= span lang=3DEN-US style=3D'font-size:12.0pt'>>
Cc: Brad Richardson = <everythingfunctional@protonmail.= com>
Subject: [EXTERNAL] [BULK] [J3] = [SC22WG5.6477] are race conditions allowed in do = concurrent?

 

=

CAUTION: This email originated from outside of NASA.  = Please take care when clicking links or opening attachments.  Use = the "Report Message" button to report suspicious messages to = the = NASA SOC.

<= p class=3DMsoNormal = style=3D'mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;mar= gin-left:72.0pt'>




Hi = Jeff,

 

=

Based on " If it is defined or becomes = undefined during any iteration, it shall not be referenced, defined, or = become undefined during any other iteration.", the provide program = is invalid, because `x` becomes defined in more than one = iteration.

 

=

The way I've heard "definition" = applied to arrays is that definition of an element of an array does not = constitute definition of the variable, because it is then only partially = defined. I.e. an array is only defined once each of its elements has = been defined.

 

=

My opinion is that the above constraint = simply does not adequately describe/is still ambiguous about the case of = arrays. I think an interp would be warranted, and we should say that an = element of an array should have the same constraint as above if the = variable is shared.

 

=

Regards,

Brad

 

=

On Tue, 2023-09-12 at 14:43 +0000, Jeff = Hammond via J3 wrote:

Is this a legal program in Fortran with = defined behavior?  If it is illegal or the behavior is undefined, = where do we say that?

 

program = main

  = integer :: k

  = real :: x

  x = =3D 0.0

  = do concurrent (k=3D1:2) shared(x)

    x =3D = real(k)

  = end do

  = print*,x

end = program main

 

Is that = what the following is intended to = do?

 

“If a variable has SHARED locality, = appearances of the variable within the DO CONCURRENT construct refer to = the variable in the innermost executable construct or scoping unit that = includes the DO CONCURRENT construct. If it is defined or becomes = undefined during any iteration, it shall not be referenced, defined, or = become undefined during any other = iteration.”

 

If so, = then I am struggling to apply this to arrays.  When we say a = variable has shared locality, is the variable the array or the = element?  We can only put array names in SHARED(), but applying the = text regarding “to become defined” from the perspective of = arrays appears to have interesting = consequences.

 

Thanks,


Jeff

<= /div>

 

= ------=_NextPart_000_0473_01D9EF8E.085A2420--