Alex Coplan <[email protected]> writes:
> Hi,
>
> This is a v2 version which addresses feedback from Richard's review
> here:
>
> https://gcc.gnu.org/pipermail/gcc-patches/2023-November/637648.html
>
> I'll reply inline to address specific comments.
>
> Bootstrapped/regtested on aarch64-linux-gnu, OK for trunk?
>
> Thanks,
> Alex
>
> -- >8 --
>
> This patch overhauls the load/store pair patterns with two main goals:
>
> 1. Fixing a correctness issue (the current patterns are not RA-friendly).
> 2. Allowing more flexibility in which operand modes are supported, and which
> combinations of modes are allowed in the two arms of the load/store pair,
> while reducing the number of patterns required both in the source and in
> the generated code.
>
> The correctness issue (1) is due to the fact that the current patterns have
> two independent memory operands tied together only by a predicate on the
> insns.
> Since LRA only looks at the constraints, one of the memory operands can get
> reloaded without the other one being changed, leading to the insn becoming
> unrecognizable after reload.
>
> We fix this issue by changing the patterns such that they only ever have one
> memory operand representing the entire pair. For the store case, we use an
> unspec to logically concatenate the register operands before storing them.
> For the load case, we use unspecs to extract the "lanes" from the pair mem,
> with the second occurrence of the mem matched using a match_dup (such that
> there
> is still really only one memory operand as far as the RA is concerned).
>
> In terms of the modes used for the pair memory operands, we canonicalize
> these to V2x4QImode, V2x8QImode, and V2x16QImode. These modes have not
> only the correct size but also correct alignment requirement for a
> memory operand representing an entire load/store pair. Unlike the other
> two, V2x4QImode didn't previously exist, so had to be added with the
> patch.
>
> As with the previous patch generalizing the writeback patterns, this
> patch aims to be flexible in the combinations of modes supported by the
> patterns without requiring a large number of generated patterns by using
> distinct mode iterators.
>
> The new scheme means we only need a single (generated) pattern for each
> load/store operation of a given operand size. For the 4-byte and 8-byte
> operand cases, we use the GPI iterator to synthesize the two patterns.
> The 16-byte case is implemented as a separate pattern in the source (due
> to only having a single possible alternative).
>
> Since the UNSPEC patterns can't be interpreted by the dwarf2cfi code,
> we add REG_CFA_OFFSET notes to the store pair insns emitted by
> aarch64_save_callee_saves, so that correct CFI information can still be
> generated. Furthermore, we now unconditionally generate these CFA
> notes on frame-related insns emitted by aarch64_save_callee_saves.
> This is done in case that the load/store pair pass forms these into
> pairs, in which case the CFA notes would be needed.
>
> We also adjust the ldp/stp peepholes to generate the new form. This is
> done by switching the generation to use the
> aarch64_gen_{load,store}_pair interface, making it easier to change the
> form in the future if needed. (Likewise, the upcoming aarch64
> load/store pair pass also makes use of this interface).
>
> This patch also adds an "ldpstp" attribute to the non-writeback
> load/store pair patterns, which is used by the post-RA load/store pair
> pass to identify existing patterns and see if they can be promoted to
> writeback variants.
>
> One potential concern with using unspecs for the patterns is that it can block
> optimization by the generic RTL passes. This patch series tries to mitigate
> this in two ways:
> 1. The pre-RA load/store pair pass runs very late in the pre-RA pipeline.
> 2. A later patch in the series adjusts the aarch64 mem{cpy,set} expansion to
> emit individual loads/stores instead of ldp/stp. These should then be
> formed back into load/store pairs much later in the RTL pipeline by the
> new load/store pair pass.
>
> gcc/ChangeLog:
>
> * config/aarch64/aarch64-ldpstp.md: Abstract ldp/stp
> representation from peepholes, allowing use of new form.
> * config/aarch64/aarch64-modes.def (V2x4QImode): Define.
> * config/aarch64/aarch64-protos.h
> (aarch64_finish_ldpstp_peephole): Declare.
> (aarch64_swap_ldrstr_operands): Delete declaration.
> (aarch64_gen_load_pair): Adjust parameters.
> (aarch64_gen_store_pair): Likewise.
> * config/aarch64/aarch64-simd.md (load_pair<DREG:mode><DREG2:mode>):
> Delete.
> (vec_store_pair<DREG:mode><DREG2:mode>): Delete.
> (load_pair<VQ:mode><VQ2:mode>): Delete.
> (vec_store_pair<VQ:mode><VQ2:mode>): Delete.
> * config/aarch64/aarch64.cc (aarch64_pair_mode_for_mode): New.
> (aarch64_gen_store_pair): Adjust to use new unspec form of stp.
> Drop second mem from parameters.
> (aarch64_gen_load_pair): Likewise.
> (aarch64_pair_mem_from_base): New.
> (aarch64_save_callee_saves): Emit REG_CFA_OFFSET notes for
> frame-related saves. Adjust call to aarch64_gen_store_pair
> (aarch64_restore_callee_saves): Adjust calls to
> aarch64_gen_load_pair to account for change in interface.
> (aarch64_process_components): Likewise.
> (aarch64_classify_address): Handle 32-byte pair mems in
> LDP_STP_N case.
> (aarch64_print_operand): Likewise.
> (aarch64_copy_one_block_and_progress_pointers): Adjust calls to
> account for change in aarch64_gen_{load,store}_pair interface.
> (aarch64_set_one_block_and_progress_pointer): Likewise.
> (aarch64_finish_ldpstp_peephole): New.
> (aarch64_gen_adjusted_ldpstp): Adjust to use generation helper.
> * config/aarch64/aarch64.md (ldpstp): New attribute.
> (load_pair_sw_<SX:mode><SX2:mode>): Delete.
> (load_pair_dw_<DX:mode><DX2:mode>): Delete.
> (load_pair_dw_<TX:mode><TX2:mode>): Delete.
> (*load_pair_<ldst_sz>): New.
> (*load_pair_16): New.
> (store_pair_sw_<SX:mode><SX2:mode>): Delete.
> (store_pair_dw_<DX:mode><DX2:mode>): Delete.
> (store_pair_dw_<TX:mode><TX2:mode>): Delete.
> (*store_pair_<ldst_sz>): New.
> (*store_pair_16): New.
> (*load_pair_extendsidi2_aarch64): Adjust to use new form.
> (*zero_extendsidi2_aarch64): Likewise.
> * config/aarch64/iterators.md (VPAIR): New.
> * config/aarch64/predicates.md (aarch64_mem_pair_operand): Change to
> a special predicate derived from aarch64_mem_pair_operator.
Thanks, looks great. OK for trunk with a couple of trivial changes:
> [...]
> diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> index 8faaa748a05..c2e4f531254 100644
> --- a/gcc/config/aarch64/aarch64.cc
> +++ b/gcc/config/aarch64/aarch64.cc
> @@ -6680,59 +6680,87 @@ aarch64_pop_regs (unsigned regno1, unsigned regno2,
> HOST_WIDE_INT adjustment,
> }
> }
>
> -/* Generate and return a store pair instruction of mode MODE to store
> - register REG1 to MEM1 and register REG2 to MEM2. */
> +/* Given an ldp/stp register operand mode MODE, return a suitable mode to use
> + for a mem rtx representing the entire pair. */
>
> -static rtx
> -aarch64_gen_store_pair (machine_mode mode, rtx mem1, rtx reg1, rtx mem2,
> - rtx reg2)
> -{
> - switch (mode)
> - {
> - case E_DImode:
> - return gen_store_pair_dw_didi (mem1, reg1, mem2, reg2);
> +static machine_mode
> +aarch64_pair_mode_for_mode (machine_mode mode)
> +{
> + if (known_eq (GET_MODE_SIZE (mode), 4))
> + return V2x4QImode;
> + else if (known_eq (GET_MODE_SIZE (mode), 8))
> + return V2x8QImode;
> + else if (known_eq (GET_MODE_SIZE (mode), 16))
> + return V2x16QImode;
> + else
> + gcc_unreachable ();
> +}
>
> - case E_DFmode:
> - return gen_store_pair_dw_dfdf (mem1, reg1, mem2, reg2);
> +/* Given a base mem MEM with a mode suitable for an ldp/stp register operand,
> + return an rtx like MEM which instead represents the entire pair. */
Might be worth saying "mode and address suitable...", given the assert
for aarch64_mem_pair_lanes_operand.
>
> - case E_TFmode:
> - return gen_store_pair_dw_tftf (mem1, reg1, mem2, reg2);
> +static rtx
> +aarch64_pair_mem_from_base (rtx mem)
> +{
> + auto pair_mode = aarch64_pair_mode_for_mode (GET_MODE (mem));
> + mem = adjust_bitfield_address_nv (mem, pair_mode, 0);
> + gcc_assert (aarch64_mem_pair_lanes_operand (mem, pair_mode));
> + return mem;
> +}
>
> - case E_V4SImode:
> - return gen_vec_store_pairv4siv4si (mem1, reg1, mem2, reg2);
> +/* Generate and return a store pair instruction to store REG1 and REG2
> + into memory starting at BASE_MEM. All three rtxes should have modes of
> the
> + same size. */
>
> - case E_V16QImode:
> - return gen_vec_store_pairv16qiv16qi (mem1, reg1, mem2, reg2);
> +rtx
> +aarch64_gen_store_pair (rtx base_mem, rtx reg1, rtx reg2)
> +{
> + rtx pair_mem = aarch64_pair_mem_from_base (base_mem);
>
> - default:
> - gcc_unreachable ();
> - }
> + return gen_rtx_SET (pair_mem,
> + gen_rtx_UNSPEC (GET_MODE (pair_mem),
> + gen_rtvec (2, reg1, reg2),
> + UNSPEC_STP));
> }
>
> -/* Generate and regurn a load pair isntruction of mode MODE to load register
> - REG1 from MEM1 and register REG2 from MEM2. */
> +/* Generate and return a load pair instruction to load a pair of
> + registers starting at BASE_MEM into REG1 and REG2. If CODE is
> + UNKNOWN, all three rtxes should have modes of the same size.
> + Otherwise, CODE is {SIGN,ZERO}_EXTEND, base_mem should be in SImode,
> + and REG{1,2} should be in DImode. */
>
> -static rtx
> -aarch64_gen_load_pair (machine_mode mode, rtx reg1, rtx mem1, rtx reg2,
> - rtx mem2)
> +rtx
> +aarch64_gen_load_pair (rtx reg1, rtx reg2, rtx base_mem, enum rtx_code code)
> {
> - switch (mode)
> - {
> - case E_DImode:
> - return gen_load_pair_dw_didi (reg1, mem1, reg2, mem2);
> -
> - case E_DFmode:
> - return gen_load_pair_dw_dfdf (reg1, mem1, reg2, mem2);
> + rtx pair_mem = aarch64_pair_mem_from_base (base_mem);
>
> - case E_TFmode:
> - return gen_load_pair_dw_tftf (reg1, mem1, reg2, mem2);
> + const bool any_extend_p = (code == ZERO_EXTEND || code == SIGN_EXTEND);
> + if (any_extend_p)
> + {
> + gcc_checking_assert (GET_MODE (base_mem) == SImode
> + && GET_MODE (reg1) == DImode
> + && GET_MODE (reg2) == DImode);
> + }
No need for the braces.
> + else
> + gcc_assert (code == UNKNOWN);
> +
> + rtx unspecs[2] = {
> + gen_rtx_UNSPEC (any_extend_p ? SImode : GET_MODE (reg1),
> + gen_rtvec (1, pair_mem),
> + UNSPEC_LDP_FST),
> + gen_rtx_UNSPEC (any_extend_p ? SImode : GET_MODE (reg2),
> + gen_rtvec (1, copy_rtx (pair_mem)),
> + UNSPEC_LDP_SND)
> + };
>
> - case E_V4SImode:
> - return gen_load_pairv4siv4si (reg1, mem1, reg2, mem2);
> + if (any_extend_p)
> + for (int i = 0; i < 2; i++)
> + unspecs[i] = gen_rtx_fmt_e (code, DImode, unspecs[i]);
>
> - default:
> - gcc_unreachable ();
> - }
> + return gen_rtx_PARALLEL (VOIDmode,
> + gen_rtvec (2,
> + gen_rtx_SET (reg1, unspecs[0]),
> + gen_rtx_SET (reg2, unspecs[1])));
> }
>
> /* Return TRUE if return address signing should be enabled for the current
> [...]
> @@ -453,6 +456,11 @@ (define_attr "predicated" "yes,no" (const_string "no"))
> ;; may chose to hold the tracking state encoded in SP.
> (define_attr "speculation_barrier" "true,false" (const_string "false"))
>
> +;; Attribute use to identify load pair and store pair instructions.
used
> +;; Currently the attribute is only applied to the non-writeback ldp/stp
> +;; patterns.
> +(define_attr "ldpstp" "ldp,stp,none" (const_string "none"))
> +
> ;; -------------------------------------------------------------------
> ;; Pipeline descriptions and scheduling
> ;; -------------------------------------------------------------------
