Re: [lldb-dev] [Release-testers] [7.0.0 Release] rc3 has been tagged

[Bero Rosenkränzer via lldb-dev]

Hi,
OpenMandriva is updated and runs well, so far we've found one problem
while rebuilding the OS with rc3:

http://file-store.openmandriva.org/api/v1/file_stores/e95b2bd9e6e7dfcfbbab1906d20763879fa6f489.log?show=true

Building libqalculate on armv7hnl causes the compiler to barf. I
didn't get around to debugging this yet. Same package builds ok on all
other supported arches (aarch64, x86_64, i686).

ttyl
bero

On Mon, 10 Sep 2018 at 16:13, Hans Wennborg via Release-testers
 wrote:
>
> Dear testers,
>
> 7.0.0-rc3 was just tagged (from branch revision r341805).
>
> No further release candidates are currently planned, so this is a
> release candidate in the real sense: unless any serious issues
> surface, this is what the final release will look like.
>
> Please run the test script, share your results and upload binaries.
>
> Please also take a look at the release notes and other docs; small
> changes to those are still welcome.
>
> The sources and docs will show up at
> https://prereleases.llvm.org/7.0.0/#rc3 anytime now, and binaries will
> be posted when they're ready.
>
> Thanks everyone for your work on this release!
>
> Hans
> ___
> Release-testers mailing list
> release-test...@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/release-testers
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Extract compile unit infos from OSO entries for LTO objects

[Stefan Gränitz via lldb-dev]

Thanks for your replies!

>> Right now we assume that each .o file has only on CU so we don't need to 
>> open all .o files in SymbolFileDWARFDebugMap::CalculateAbilities() which is 
>> something that gets run when we are trying to figure out which SymbolFile 
>> plug-in to load. [...] The problem used to be that even if we had a dSYM 
>> file, the loop that selected the symbol file plug-in would give each and 
>> every symbol file plugin a chance ...

Ok makes total sense, kind of historical reason.

>> But that being said, in the past I re-ordered how the SymbolFile plug-ins 
>> were initialized to always put SymbolFileDWARF first and if it finds DWARF 
>> debug info or a dSYM file and has all the abilities then we stop looking for 
>> symbol file plug-ins that can load the current file. [...] So as soon as a 
>> SymbolFile plug-in can do everything we now stop.

Good to know!

> Note that counting the number of compile units can be done extremely cheaply 
> (of course, except the cost of opening the file). Each unit as a header that 
> contain its length (which gives you the next unit). I’m not sure we have a 
> primitive that does this without parsing the DWARF, but it should be easy to 
> add.


Right, so we only need to parse the CU headers. That should be fast.
Opening each candidate .o file for the relatively rare case of having multiple 
CUs still sounds expensive, assuming that “thousands of .o files” may actually 
happen.

CalculateAbilities() does indeed call GetNumCompileUnits(), but what it really 
wants to know at this time is “do we have any CU in there”:

uint32_t SymbolFileDWARFDebugMap::CalculateAbilities() {
  ...
  const uint32_t oso_index_count = GetNumCompileUnits();
  if (oso_index_count > 0) {
InitOSO();
if (!m_compile_unit_infos.empty()) {
  return SymbolFile::CompileUnits | SymbolFile::Functions | …;
  }
}

As far as I can tell, we need the actual number of CUs only after we discovered 
plugins. In my case it’s during breakpoint resolution (i.e. 
BreakpointResolverFileLine::SearchCallback()). If we separated these two 
concerns conceptually (into HasCompileUnits() and GetNumCompileUnits()), 
couldn’t we then also do InitOSO() in two steps? Especially since lazy init is 
used everywhere already. This would avoid impact on CalculateAbilities() 
entirely.

That said, I don’t really know how big the change would get then. And it 
probably adds complexity, while the implementation is quite complex already.
Anyway, for now what do you think about the idea?


> Do we really need this CU <-> Symbol mapping?

It’s used in SymbolFileDWARFDebugMap::SymbolContainsSymbolWithIndex(), which 
looks like dead code.
It’s also used in SymbolFileDWARFDebugMap::CompileUnitInfo::GetFileRangeMap(), 
which initialises the OSO range map once. In order to do that it iterates over 
all CUs, so changing this or adding a special case here seems possible.

https://github.com/llvm-mirror/lldb/blob/59608853be9b52d3c01609196d152b3e3dbb4dac/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.cpp#L172

What do you think?

Best
Stefan

> On 11. Sep 2018, at 18:03, Frédéric Riss  wrote:
> 
> 
>> On Sep 11, 2018, at 8:48 AM, Greg Clayton > > wrote:
>> 
>>> 
>>> On Sep 11, 2018, at 2:55 AM, Stefan Gränitz >> > wrote:
>>> 
>>> Hello everyone
>>> 
>>> I am investigating a bug that prevents correct breakpoint resolution in LTO 
>>> objects with embedded DWARF (no separate dSYM file) and tracked it down to 
>>> the initialization of SymbolFileDWARFDebugMap. This code seems to assume 
>>> there is only one compile unit per object file, but LTO objects have more 
>>> than that:
>>> 
>>> void SymbolFileDWARFDebugMap::InitOSO() {
>>> ...
>>>   const uint32_t oso_index_count =
>>> symtab->AppendSymbolIndexesWithTypeAndFlagsValue(
>>> eSymbolTypeObjectFile, k_oso_symbol_flags_value, oso_indexes);
>>> ...
>>>   m_compile_unit_infos.resize(oso_index_count); // <—— one CU per OSO entry 
>>> in the Symtab
>>> 
>>>   for (uint32_t i = 0; i < oso_index_count; ++i) {
>>> const uint32_t so_idx = oso_indexes[i] - 1;
>>> const uint32_t oso_idx = oso_indexes[i];
>>> const Symbol *so_symbol = symtab->SymbolAtIndex(so_idx);
>>> const Symbol *oso_symbol = symtab->SymbolAtIndex(oso_idx);
>>> ...
>>>   const Symbol *last_symbol = symtab->SymbolAtIndex(sibling_idx - 1);
>>>   m_compile_unit_infos[i].first_symbol_index = so_idx;
>>>   m_compile_unit_infos[i].last_symbol_index = sibling_idx - 1;
>>>   m_compile_unit_infos[i].first_symbol_id = so_symbol->GetID();
>>>   m_compile_unit_infos[i].last_symbol_id = last_symbol->GetID();
>>> 
>>> The symptom is that LLDB will only read debug_line for one CU and miss all 
>>> the rest. Thus, breakpoints in other CUs can’t be associated with line 
>>> information.
>>> 
>>> I wonder if there is a good way to populate the correct number of compile 
>>> units from the OSO entry at this 

[lldb-dev] Large core files causing issues in LLDB

[Greg Clayton via lldb-dev]

In LLDB we try to mmap any object file we load including core files. Recently I 
received a 468GB core file. mmap fails in LLVM and then it will fall back to 
trying to allocate a memory buffer that large and reading a file into it.

The failing call occurs in:

DataBufferSP ObjectFile::MapFileData(const FileSpec , uint64_t Size,
 uint64_t Offset) {
  return DataBufferLLVM::CreateSliceFromPath(file.GetPath(), Size, Offset);
}

Does LLVM have any class that could help deal with really large files and 
randomly accessing data within it?
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [Release-testers] [7.0.0 Release] rc3 has been tagged

[Michał Górny via lldb-dev]

On Mon, 2018-09-10 at 16:12 +0200, Hans Wennborg via Release-testers
wrote:
> Dear testers,
> 
> 7.0.0-rc3 was just tagged (from branch revision r341805).
> 
> No further release candidates are currently planned, so this is a
> release candidate in the real sense: unless any serious issues
> surface, this is what the final release will look like.
> 
> Please run the test script, share your results and upload binaries.
> 

I see two regressions in compiler-rt since RC2:

Failing Tests (2):
LeakSanitizer-AddressSanitizer-i386 :: TestCases/Linux/fork_and_leak.cc
LeakSanitizer-Standalone-i386 :: TestCases/Linux/fork_and_leak.cc

Besides that, nothing new to report.

-- 
Best regards,
Michał Górny


signature.asc
Description: This is a digitally signed message part
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Extract compile unit infos from OSO entries for LTO objects

[Frédéric Riss via lldb-dev]

> On Sep 11, 2018, at 8:48 AM, Greg Clayton  wrote:
> 
>> 
>> On Sep 11, 2018, at 2:55 AM, Stefan Gränitz > > wrote:
>> 
>> Hello everyone
>> 
>> I am investigating a bug that prevents correct breakpoint resolution in LTO 
>> objects with embedded DWARF (no separate dSYM file) and tracked it down to 
>> the initialization of SymbolFileDWARFDebugMap. This code seems to assume 
>> there is only one compile unit per object file, but LTO objects have more 
>> than that:
>> 
>> void SymbolFileDWARFDebugMap::InitOSO() {
>> ...
>>   const uint32_t oso_index_count =
>> symtab->AppendSymbolIndexesWithTypeAndFlagsValue(
>> eSymbolTypeObjectFile, k_oso_symbol_flags_value, oso_indexes);
>> ...
>>   m_compile_unit_infos.resize(oso_index_count); // <—— one CU per OSO entry 
>> in the Symtab
>> 
>>   for (uint32_t i = 0; i < oso_index_count; ++i) {
>> const uint32_t so_idx = oso_indexes[i] - 1;
>> const uint32_t oso_idx = oso_indexes[i];
>> const Symbol *so_symbol = symtab->SymbolAtIndex(so_idx);
>> const Symbol *oso_symbol = symtab->SymbolAtIndex(oso_idx);
>> ...
>>   const Symbol *last_symbol = symtab->SymbolAtIndex(sibling_idx - 1);
>>   m_compile_unit_infos[i].first_symbol_index = so_idx;
>>   m_compile_unit_infos[i].last_symbol_index = sibling_idx - 1;
>>   m_compile_unit_infos[i].first_symbol_id = so_symbol->GetID();
>>   m_compile_unit_infos[i].last_symbol_id = last_symbol->GetID();
>> 
>> The symptom is that LLDB will only read debug_line for one CU and miss all 
>> the rest. Thus, breakpoints in other CUs can’t be associated with line 
>> information.
>> 
>> I wonder if there is a good way to populate the correct number of compile 
>> units from the OSO entry at this point?
> 
> The reason it is like this is we don't want to have to open all .o files when 
> we parse the debug map in order to figure out a compile unit index. Right now 
> the compile unit UserID is just the index of the .o file in the debug map. 
> Opening thousands of .o files can impose a performance issue.
>> 
>> The situation may appear similar to an archive file with a number of 
>> objects, but then we have separate OSO entries like 
>> “path/to/lib/libLLVMMCParser.a(AsmLexer.cpp.o)”. Furthermore LTO objects 
>> have one common symtab for all compile units and it was probably mixed up by 
>> optimization, so we cannot simply say that CUs start/end at certain symbol 
>> indexes as in the above code. The latter is used rarely and only in 
>> SymbolFileDWARFDebugMap, so there may be a workaround, but first I have to 
>> figure out the initial question:
>> 
>> How to get more information about compile units in an LTO object? Any ideas 
>> welcome!
> 
> The only way is to open each .o file and see how many compile units they 
> contain. Right now we assume that each .o file has only on CU so we don't 
> need to open all .o files in SymbolFileDWARFDebugMap::CalculateAbilities() 
> which is something that gets run when we are trying to figure out which 
> SymbolFile plug-in to load. But that being said, in the past I re-ordered how 
> the SymbolFile plug-ins were initialized to always put SymbolFileDWARF first 
> and if it finds DWARF debug info or a dSYM file and has all the abilities 
> then we stop looking for symbol file plug-ins that can load the current file. 
> The problem used to be that even if we had a dSYM file, the loop that 
> selected the symbol file plug-in would give each and every symbol file plugin 
> a chance to tell us how much info they could extract via a call to 
> SymbolFile::CalculateAbilities() and that would cause us to open all .o files 
> just to say "I parse all debug info" just like a previous plug-in could. So 
> as soon as a SymbolFile plug-in can do everything we now stop.
> 
>> If that’s not possible, I may find another way to fix it further down the 
>> road, but then the name m_compile_unit_infos seems not exactly correct here. 
>> It’s rather something like m_referenced_object_infos, right?
> 
> So now that that change has been in for a while, it might be ok to open each 
> .o file and see how many compile units they contain and then populate 
> m_compile_unit_infos as needed.

Note that counting the number of compile units can be done extremely cheaply 
(of course, except the cost of opening the file). Each unit as a header that 
contain its length (which gives you the next unit). I’m not sure we have a 
primitive that does this without parsing the DWARF, but it should be easy to 
add. 

> You will need to watch for any usage of m_compile_unit_infos and make sure it 
> does the correct thing.

That’s the part I was worried about. The structure of m_compile_unit_infos 
makes the assumption that we can associate slices of symbols to a compile unit. 
I don’t think this is a correct assumption to make in the LTO case, and even if 
it were, we’d need to parse the DWARF and do some pretty heavy processing to 
extract the information. Do 

Re: [lldb-dev] Extract compile unit infos from OSO entries for LTO objects

[Greg Clayton via lldb-dev]

> On Sep 11, 2018, at 2:55 AM, Stefan Gränitz  wrote:
> 
> Hello everyone
> 
> I am investigating a bug that prevents correct breakpoint resolution in LTO 
> objects with embedded DWARF (no separate dSYM file) and tracked it down to 
> the initialization of SymbolFileDWARFDebugMap. This code seems to assume 
> there is only one compile unit per object file, but LTO objects have more 
> than that:
> 
> void SymbolFileDWARFDebugMap::InitOSO() {
> ...
>   const uint32_t oso_index_count =
> symtab->AppendSymbolIndexesWithTypeAndFlagsValue(
> eSymbolTypeObjectFile, k_oso_symbol_flags_value, oso_indexes);
> ...
>   m_compile_unit_infos.resize(oso_index_count); // <—— one CU per OSO entry 
> in the Symtab
> 
>   for (uint32_t i = 0; i < oso_index_count; ++i) {
> const uint32_t so_idx = oso_indexes[i] - 1;
> const uint32_t oso_idx = oso_indexes[i];
> const Symbol *so_symbol = symtab->SymbolAtIndex(so_idx);
> const Symbol *oso_symbol = symtab->SymbolAtIndex(oso_idx);
> ...
>   const Symbol *last_symbol = symtab->SymbolAtIndex(sibling_idx - 1);
>   m_compile_unit_infos[i].first_symbol_index = so_idx;
>   m_compile_unit_infos[i].last_symbol_index = sibling_idx - 1;
>   m_compile_unit_infos[i].first_symbol_id = so_symbol->GetID();
>   m_compile_unit_infos[i].last_symbol_id = last_symbol->GetID();
> 
> The symptom is that LLDB will only read debug_line for one CU and miss all 
> the rest. Thus, breakpoints in other CUs can’t be associated with line 
> information.
> 
> I wonder if there is a good way to populate the correct number of compile 
> units from the OSO entry at this point?

The reason it is like this is we don't want to have to open all .o files when 
we parse the debug map in order to figure out a compile unit index. Right now 
the compile unit UserID is just the index of the .o file in the debug map. 
Opening thousands of .o files can impose a performance issue.
> 
> The situation may appear similar to an archive file with a number of objects, 
> but then we have separate OSO entries like 
> “path/to/lib/libLLVMMCParser.a(AsmLexer.cpp.o)”. Furthermore LTO objects have 
> one common symtab for all compile units and it was probably mixed up by 
> optimization, so we cannot simply say that CUs start/end at certain symbol 
> indexes as in the above code. The latter is used rarely and only in 
> SymbolFileDWARFDebugMap, so there may be a workaround, but first I have to 
> figure out the initial question:
> 
> How to get more information about compile units in an LTO object? Any ideas 
> welcome!

The only way is to open each .o file and see how many compile units they 
contain. Right now we assume that each .o file has only on CU so we don't need 
to open all .o files in SymbolFileDWARFDebugMap::CalculateAbilities() which is 
something that gets run when we are trying to figure out which SymbolFile 
plug-in to load. But that being said, in the past I re-ordered how the 
SymbolFile plug-ins were initialized to always put SymbolFileDWARF first and if 
it finds DWARF debug info or a dSYM file and has all the abilities then we stop 
looking for symbol file plug-ins that can load the current file. The problem 
used to be that even if we had a dSYM file, the loop that selected the symbol 
file plug-in would give each and every symbol file plugin a chance to tell us 
how much info they could extract via a call to SymbolFile::CalculateAbilities() 
and that would cause us to open all .o files just to say "I parse all debug 
info" just like a previous plug-in could. So as soon as a SymbolFile plug-in 
can do everything we now stop.

> If that’s not possible, I may find another way to fix it further down the 
> road, but then the name m_compile_unit_infos seems not exactly correct here. 
> It’s rather something like m_referenced_object_infos, right?

So now that that change has been in for a while, it might be ok to open each .o 
file and see how many compile units they contain and then populate 
m_compile_unit_infos as needed. You will need to watch for any usage of 
m_compile_unit_infos and make sure it does the correct thing.


> Btw.: My first attempt was a workaround for the symptom (see 
> https://reviews.llvm.org/D51546 ). It simply 
> reads all debug_lines for a single CU, but I’d really appreciate a better 
> solution.

The fix in D51546 seems wrong because the only way we get to a line table is 
via the DW_AT_stmt_list from a compile unit. If we can fix the LTO case to load 
all compile units from the LTO.o files with multiple CU's this fix won't be 
needed.

So the correct solution is to detect how many compile units are in each .o file 
and then make sure to find all places that were assuming anything about the OSO 
index being the compile unit UserID are fixed. Now that plug-in loading stops 
after a SymbolFile says it can handle everything we can probably do a bit more 
work. One issue is that .o files might have been cleaned 

Re: [lldb-dev] [Release-testers] [7.0.0 Release] rc3 has been tagged

[Dimitry Andric via lldb-dev]

On 10 Sep 2018, at 16:12, Hans Wennborg via Release-testers 
 wrote:
> 7.0.0-rc3 was just tagged (from branch revision r341805).
> 
> No further release candidates are currently planned, so this is a
> release candidate in the real sense: unless any serious issues
> surface, this is what the final release will look like.

Main test results on amd64-freebsd11 look slightly better, 2 less unexpected 
failures:

  Expected Passes: 52422(rc2: 52409)
  Expected Failures  :   232(rc2:   232)
  Unsupported Tests  :  3687(rc2:  3687)
  Unexpected Passes  : 1(rc2: 1)
  Unexpected Failures:   489(rc2:   491)

Test-suite test results on amd64-freebsd11 are also better, 58 less unexpected 
failures:

  Expected Passes:   903(rc2:   845)
  Unexpected Failures: 3(rc2:61)

Test results on i386-freebsd11 were slightly worse, due to a bunch of hanging 
lldb single step tests (these all seem to hang in the STOP state indefinitely, 
and so had to be killed off):

  Expected Passes: 50226(rc2: 50186)
  Expected Failures  :   226(rc2:   226)
  Unsupported Tests  :  2502(rc2:  2502)
  Unexpected Failures:   277(rc2:   306)

The test-suite still doesn't build on i386-freebsd, but that is a known issue.

I have uploaded:

SHA256 (clang+llvm-7.0.0-rc3-amd64-unknown-freebsd11.tar.xz) = 
1b5d72f94e4f18713393ad0ce4f1509ee13b8b41fde114e9a18b2247e2a740cb
SHA256 (clang+llvm-7.0.0-rc3-i386-unknown-freebsd11.tar.xz) = 
4e09e6a9d06982ba35b6e213d74498c9b46b01bfd044728d6f81de50791dbd4a

-Dimitry



signature.asc
Description: Message signed with OpenPGP
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [llvm-dev] [7.0.0 Release] rc3 has been tagged

[Hans Wennborg via lldb-dev]

Updated now. Thanks for the reminder!

On Tue, Sep 11, 2018 at 5:08 AM, Jan Vesely  wrote:
> Hi,
>
> do you mind updating the llvm.org webpage as well? It still states the old
> schedule of final release on Sept 5.
>
> thanks,
> Jan
>
> On Mon, Sep 10, 2018 at 10:13 AM Hans Wennborg via llvm-dev
>  wrote:
>>
>> Dear testers,
>>
>> 7.0.0-rc3 was just tagged (from branch revision r341805).
>>
>> No further release candidates are currently planned, so this is a
>> release candidate in the real sense: unless any serious issues
>> surface, this is what the final release will look like.
>>
>> Please run the test script, share your results and upload binaries.
>>
>> Please also take a look at the release notes and other docs; small
>> changes to those are still welcome.
>>
>> The sources and docs will show up at
>> https://prereleases.llvm.org/7.0.0/#rc3 anytime now, and binaries will
>> be posted when they're ready.
>>
>> Thanks everyone for your work on this release!
>>
>> Hans
>> ___
>> LLVM Developers mailing list
>> llvm-...@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev