Re: [swift-dev] External pass pipeline YAML format

[Michael Gottesman via swift-dev]

> On Jun 29, 2017, at 12:50 PM, Vasileios Kalintiris  wrote:
> 
> I plan on writing a patch for this. Based on which branch should I create the 
> pull request? Is the swift-3.1-branch frozen?

master.

> 
> On Thu, Jun 29, 2017 at 7:27 PM, Michael Gottesman  > wrote:
> 
>> On Jun 28, 2017, at 10:15 AM, Vasileios Kalintiris via swift-dev 
>> > wrote:
>> 
>> > Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it 
>> > looks like it might give some hints?
>> 
>> I've tried generating a pass pipeline with 
>> utils/pass-pipeline/scripts/pipelines_generator.py and use it with the 
>> -external-pass-pipeline-filename. However, AFAICT the generated pipeline is 
>> not in the correct format that the compiler expects:
>> 
>> [
>> [
>> "HighLevel",
>> "run_n_times",
>> 2,
>> "SimplifyCFG",
>> ...
>> "GlobalARCOpts"
>> ],
>> [
>> "EarlyLoopOpt",
>> "run_n_times",
>> 1,
>> "LowerAggregateInstrs",
>> ...
>> "SwiftArrayOpts"
>> ],
>> ...
>> ]
>> 
>> Each generated pass pipeline contains the "run_n_times"|"run_to_fixed_point" 
>> field, followed by the number of iterations, which is not what the compiler 
>> expects.
>> 
>> I had no luck even when I tried to re-format the file containing the 
>> pipelines to something that I believe the compiler would expect based on the 
>> source code of SILPassPipelinePlan::getPassPipelineFromFile():
>> 
>> [
>> [
>> "HihLevel",
>> "SimplifyCFG",
>> ...
>> "GlobalARCOpts"
>> ],
>> ...
>> ]
>> 
>> or even:
>> 
>> [
>> [
>>"HighLevel",
>>[ "SimplifyCFG" ],
>>...
>>[ "GlobalARCOpts" ]
>> ],
>> ...
>> ]
>> 
>> I suspect that we don't use the pass pipeline python scripts in our 
>> buildbots anymore and the relevant bits, ie. the code in 
>> SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts, have 
>> not been kept up-to-date. 
> 
> Yes. I think that is true. Here is what I would suggest:
> 
> 1. It would be really trivial to change this to use yamltraits.
> 2. If you make this change, make sure that a test is added (maybe to sil-opt) 
> that makes sure that we can roundtrip from the dumper.
> 
> MIchael
> 
>> 
>> 
>> On Wed, Jun 28, 2017 at 6:18 PM, Daniel Dunbar > > wrote:
>> Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it 
>> looks like it might give some hints?
>> --
>> $ sgit grep external-pass-pipeline-filename
>> include/swift/Option/FrontendOptions.td:def external_pass_pipeline_filename 
>> : Separate<["-"], "external-pass-pipeline-filename">,
>> utils/pass-pipeline/scripts/pipelines_build_script.py:
>> '-external-pass-pipeline-filename\;-Xfrontend\;%s' % data_file]
>> --
>> 
>>  - Daniel
>> 
>> > On Jun 28, 2017, at 5:27 AM, Vasileios Kalintiris via swift-dev 
>> > > wrote:
>> >
>> > Hi all,
>> >
>> > Please, let me know if I should post this to another list.
>> >
>> > I'm trying to figure out what is the expected YAML format of the 
>> > -external-pass-pipeline-filename option.
>> >
>> > Dumping the pass pipeline under -O and feeding it back to the compiler 
>> > with this option doesn't work (from swift-3.1-branch).
>> >
>> > I thought to ask here because I'm not entirely sure that the relevant YAML 
>> > parsing code from SILPassPipelinePlan::getPassPipelineFromFile() accepts 
>> > valid YAML input.
>> >
>> > Thanks,
>> > Vasileios
>> > ___
>> > swift-dev mailing list
>> > swift-dev@swift.org 
>> > https://lists.swift.org/mailman/listinfo/swift-dev 
>> > 
>> 
>> 
>> ___
>> swift-dev mailing list
>> swift-dev@swift.org 
>> https://lists.swift.org/mailman/listinfo/swift-dev 
>> 
> 
> 

___
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Re: [swift-dev] External pass pipeline YAML format

[Michael Gottesman via swift-dev]

> On Jun 28, 2017, at 10:15 AM, Vasileios Kalintiris via swift-dev 
>  wrote:
> 
> > Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it 
> > looks like it might give some hints?
> 
> I've tried generating a pass pipeline with 
> utils/pass-pipeline/scripts/pipelines_generator.py and use it with the 
> -external-pass-pipeline-filename. However, AFAICT the generated pipeline is 
> not in the correct format that the compiler expects:
> 
> [
> [
> "HighLevel",
> "run_n_times",
> 2,
> "SimplifyCFG",
> ...
> "GlobalARCOpts"
> ],
> [
> "EarlyLoopOpt",
> "run_n_times",
> 1,
> "LowerAggregateInstrs",
> ...
> "SwiftArrayOpts"
> ],
> ...
> ]
> 
> Each generated pass pipeline contains the "run_n_times"|"run_to_fixed_point" 
> field, followed by the number of iterations, which is not what the compiler 
> expects.
> 
> I had no luck even when I tried to re-format the file containing the 
> pipelines to something that I believe the compiler would expect based on the 
> source code of SILPassPipelinePlan::getPassPipelineFromFile():
> 
> [
> [
> "HihLevel",
> "SimplifyCFG",
> ...
> "GlobalARCOpts"
> ],
> ...
> ]
> 
> or even:
> 
> [
> [
>"HighLevel",
>[ "SimplifyCFG" ],
>...
>[ "GlobalARCOpts" ]
> ],
> ...
> ]
> 
> I suspect that we don't use the pass pipeline python scripts in our buildbots 
> anymore and the relevant bits, ie. the code in 
> SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts, have 
> not been kept up-to-date. 

Yes. I think that is true. Here is what I would suggest:

1. It would be really trivial to change this to use yamltraits.
2. If you make this change, make sure that a test is added (maybe to sil-opt) 
that makes sure that we can roundtrip from the dumper.

MIchael

> 
> 
> On Wed, Jun 28, 2017 at 6:18 PM, Daniel Dunbar  > wrote:
> Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it 
> looks like it might give some hints?
> --
> $ sgit grep external-pass-pipeline-filename
> include/swift/Option/FrontendOptions.td:def external_pass_pipeline_filename : 
> Separate<["-"], "external-pass-pipeline-filename">,
> utils/pass-pipeline/scripts/pipelines_build_script.py:
> '-external-pass-pipeline-filename\;-Xfrontend\;%s' % data_file]
> --
> 
>  - Daniel
> 
> > On Jun 28, 2017, at 5:27 AM, Vasileios Kalintiris via swift-dev 
> > > wrote:
> >
> > Hi all,
> >
> > Please, let me know if I should post this to another list.
> >
> > I'm trying to figure out what is the expected YAML format of the 
> > -external-pass-pipeline-filename option.
> >
> > Dumping the pass pipeline under -O and feeding it back to the compiler with 
> > this option doesn't work (from swift-3.1-branch).
> >
> > I thought to ask here because I'm not entirely sure that the relevant YAML 
> > parsing code from SILPassPipelinePlan::getPassPipelineFromFile() accepts 
> > valid YAML input.
> >
> > Thanks,
> > Vasileios
> > ___
> > swift-dev mailing list
> > swift-dev@swift.org 
> > https://lists.swift.org/mailman/listinfo/swift-dev 
> > 
> 
> 
> ___
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev

___
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Re: [swift-dev] External pass pipeline YAML format

[Vasileios Kalintiris via swift-dev]

> I suspect that we don't use the pass pipeline python scripts in our
buildbots anymore and the relevant bits, ie. the code in
SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts,
have not been kept up-to-date.

It seems that the following format works for the time being:

[
  [ "Pipeline1", [ "PassName1" ],
[ "Pipeline2", [ "PassName2" ],
  [ "Pipeline3", [ "PassName3" ],
[ ...
]
  ]
]
  ]
]

However, it works only when LLVM's assertions are disabled. I'll try to
come-up with a patch that will accept a saner format with multiple passes
per pipeline entry. Against which branch should I perform a pull request?

- Vasileios



On Wed, Jun 28, 2017 at 7:15 PM, Vasileios Kalintiris  wrote:

> > Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`,
> it looks like it might give some hints?
>
> I've tried generating a pass pipeline with 
> utils/pass-pipeline/scripts/pipelines_generator.py
> and use it with the -external-pass-pipeline-filename. However, AFAICT the
> generated pipeline is not in the correct format that the compiler expects:
>
> [
> [
> "HighLevel",
> "run_n_times",
> 2,
> "SimplifyCFG",
> ...
> "GlobalARCOpts"
> ],
> [
> "EarlyLoopOpt",
> "run_n_times",
> 1,
> "LowerAggregateInstrs",
> ...
> "SwiftArrayOpts"
> ],
> ...
> ]
>
> Each generated pass pipeline contains the "run_n_times"|"run_to_fixed_point"
> field, followed by the number of iterations, which is not what the compiler
> expects.
>
> I had no luck even when I tried to re-format the file containing the
> pipelines to something that I believe the compiler would expect based on
> the source code of SILPassPipelinePlan::getPassPipelineFromFile():
>
> [
> [
> "HihLevel",
> "SimplifyCFG",
> ...
> "GlobalARCOpts"
> ],
> ...
> ]
>
> or even:
>
> [
> [
>"HighLevel",
>[ "SimplifyCFG" ],
>...
>[ "GlobalARCOpts" ]
> ],
> ...
> ]
>
> I suspect that we don't use the pass pipeline python scripts in our
> buildbots anymore and the relevant bits, ie. the code in
> SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts,
> have not been kept up-to-date.
>
>
> On Wed, Jun 28, 2017 at 6:18 PM, Daniel Dunbar 
> wrote:
>
>> Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`,
>> it looks like it might give some hints?
>> --
>> $ sgit grep external-pass-pipeline-filename
>> include/swift/Option/FrontendOptions.td:def
>> external_pass_pipeline_filename : Separate<["-"],
>> "external-pass-pipeline-filename">,
>> utils/pass-pipeline/scripts/pipelines_build_script.py:
>> '-external-pass-pipeline-filename\;-Xfrontend\;%s' % data_file]
>> --
>>
>>  - Daniel
>>
>> > On Jun 28, 2017, at 5:27 AM, Vasileios Kalintiris via swift-dev <
>> swift-dev@swift.org> wrote:
>> >
>> > Hi all,
>> >
>> > Please, let me know if I should post this to another list.
>> >
>> > I'm trying to figure out what is the expected YAML format of the
>> -external-pass-pipeline-filename option.
>> >
>> > Dumping the pass pipeline under -O and feeding it back to the compiler
>> with this option doesn't work (from swift-3.1-branch).
>> >
>> > I thought to ask here because I'm not entirely sure that the relevant
>> YAML parsing code from SILPassPipelinePlan::getPassPipelineFromFile()
>> accepts valid YAML input.
>> >
>> > Thanks,
>> > Vasileios
>> > ___
>> > swift-dev mailing list
>> > swift-dev@swift.org
>> > https://lists.swift.org/mailman/listinfo/swift-dev
>>
>>
>
___
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Re: [swift-dev] External pass pipeline YAML format

[Vasileios Kalintiris via swift-dev]

> Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it
looks like it might give some hints?

I've tried generating a pass pipeline with
utils/pass-pipeline/scripts/pipelines_generator.py
and use it with the -external-pass-pipeline-filename. However, AFAICT the
generated pipeline is not in the correct format that the compiler expects:

[
[
"HighLevel",
"run_n_times",
2,
"SimplifyCFG",
...
"GlobalARCOpts"
],
[
"EarlyLoopOpt",
"run_n_times",
1,
"LowerAggregateInstrs",
...
"SwiftArrayOpts"
],
...
]

Each generated pass pipeline contains the "run_n_times"|"run_to_fixed_point"
field, followed by the number of iterations, which is not what the compiler
expects.

I had no luck even when I tried to re-format the file containing the
pipelines to something that I believe the compiler would expect based on
the source code of SILPassPipelinePlan::getPassPipelineFromFile():

[
[
"HihLevel",
"SimplifyCFG",
...
"GlobalARCOpts"
],
...
]

or even:

[
[
   "HighLevel",
   [ "SimplifyCFG" ],
   ...
   [ "GlobalARCOpts" ]
],
...
]

I suspect that we don't use the pass pipeline python scripts in our
buildbots anymore and the relevant bits, ie. the code in
SILPassPipelinePlan::getPassPipelineFromFile and/or the python scripts,
have not been kept up-to-date.


On Wed, Jun 28, 2017 at 6:18 PM, Daniel Dunbar 
wrote:

> Have you seen `utils/pass-pipeline/scripts/pipelines_build_script.py`, it
> looks like it might give some hints?
> --
> $ sgit grep external-pass-pipeline-filename
> include/swift/Option/FrontendOptions.td:def external_pass_pipeline_filename
> : Separate<["-"], "external-pass-pipeline-filename">,
> utils/pass-pipeline/scripts/pipelines_build_script.py:
> '-external-pass-pipeline-filename\;-Xfrontend\;%s' % data_file]
> --
>
>  - Daniel
>
> > On Jun 28, 2017, at 5:27 AM, Vasileios Kalintiris via swift-dev <
> swift-dev@swift.org> wrote:
> >
> > Hi all,
> >
> > Please, let me know if I should post this to another list.
> >
> > I'm trying to figure out what is the expected YAML format of the
> -external-pass-pipeline-filename option.
> >
> > Dumping the pass pipeline under -O and feeding it back to the compiler
> with this option doesn't work (from swift-3.1-branch).
> >
> > I thought to ask here because I'm not entirely sure that the relevant
> YAML parsing code from SILPassPipelinePlan::getPassPipelineFromFile()
> accepts valid YAML input.
> >
> > Thanks,
> > Vasileios
> > ___
> > swift-dev mailing list
> > swift-dev@swift.org
> > https://lists.swift.org/mailman/listinfo/swift-dev
>
>
___
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev