Re: Problem with batch logging
Hi, Mike, Ehsan, and David, thank you! In the end, I added the Logging functionality to mozglue, exported it, and imported it wherever necessary. Now it works as expected, and there are no noticeable hangings even when working with large applications (e.g. facebook). Just in case, if a beginner like me, runs into a similar problem, this link was also useful: http://msdn.microsoft.com/en-us/library/ms235636.aspx Thank you, Josip On Wednesday, October 29, 2014 12:19:03 AM UTC+1, Mike Hommey wrote: On Wed, Oct 29, 2014 at 08:08:23AM +0900, Mike Hommey wrote: On Tue, Oct 28, 2014 at 11:56:49AM -0400, Ehsan Akhgari wrote: On 2014-10-28 4:16 AM, Josip Maras wrote: Hi Ehsan, Yes, in my opinion that is the problem. I'm trying to use the global stream variable across module boundaries, more specifically from the following source files: content\base\src\Element.cpp; layout\style\nsCSSParser.cpp; content\base\src\nsINode.cpp, js\src\builtin\Eval.cpp, js\src\vm\Interpreter.cpp, js\src\jsfun.cpp, browser\app\nsBrowserApp.cpp, content\base\src\nsDocument.cpp, dom\base\nsGlobalWindow.cpp, parser\html\nsHtml5TreeOperation.cpp, content\html\document\src\nsHTMLContentSink.cpp So it's across at least two different modules. I've tried to follow your advice in using MOZ_EXPORT and MOZ_IMPORT, but I'm not getting anywhere :-/ The application runs, but it crashes. Of the top of your head, do you maybe know of a variable that is used in a similar way, so that I can look into how this is supposed to be done. For example, if I wanted to create a stream in nsBrowserApp.cpp and then use that stream from the other files what should I do? Do I put: extern MOZ_EXPORT std::stringstream FC_LOG_STREAM; in that file (or in the header included from that file, and then i define the varible in the .cpp file) and then use MOZ_IMPORT std::stringstream FC_LOG_STREAM; in the other files/headers or? Sorry for taking up your time, but getting into the source code of Firefox, especially since I've done very little real world development in C++ is a bit overwhelming. Any help will be appreciated! These macros just expand to __declspec(dllimport/dllexport), which is documented here: http://msdn.microsoft.com/en-us/library/3y1sfaz2.aspx Hopefully you can fix your problem following those instructions. Note that due to the nature of how we load xul.dll (which is done dynamically at runtime), I think your best bet is to export the variable from firefox.exe and import it in xul.dll, otherwise firefox.exe will not load because it will try to import a symbol from xul.dll but that DLL does not exist at startup time. It's not possible to use a symbol in an executable from a dll. Not directly. The best place for something that needs to be shared between firefox.exe and xul.dll is mozglue.dll. Both are linked against it. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Problem with batch logging
Hi Ehsan,
Yes, in my opinion that is the problem. I'm trying to use the global stream
variable across module boundaries, more specifically from the following source
files: content\base\src\Element.cpp; layout\style\nsCSSParser.cpp;
content\base\src\nsINode.cpp, js\src\builtin\Eval.cpp,
js\src\vm\Interpreter.cpp, js\src\jsfun.cpp, browser\app\nsBrowserApp.cpp,
content\base\src\nsDocument.cpp, dom\base\nsGlobalWindow.cpp,
parser\html\nsHtml5TreeOperation.cpp,
content\html\document\src\nsHTMLContentSink.cpp
So it's across at least two different modules.
I've tried to follow your advice in using MOZ_EXPORT and MOZ_IMPORT, but I'm
not getting anywhere :-/ The application runs, but it crashes.
Of the top of your head, do you maybe know of a variable that is used in a
similar way, so that I can look into how this is supposed to be done.
For example, if I wanted to create a stream in nsBrowserApp.cpp and then use
that stream from the other files what should I do?
Do I put:
extern MOZ_EXPORT std::stringstream FC_LOG_STREAM; in that file (or in the
header included from that file, and then i define the varible in the .cpp file)
and then use MOZ_IMPORT std::stringstream FC_LOG_STREAM; in the other
files/headers or?
Sorry for taking up your time, but getting into the source code of Firefox,
especially since I've done very little real world development in C++ is a bit
overwhelming. Any help will be appreciated!
Thank you
On Friday, October 24, 2014 7:26:00 PM UTC+2, Ehsan Akhgari wrote:
It's hard to determine exactly what's happening because your post
doesn't explain where you're using this variable, but note that extern
variables cannot be used across module boundaries. For example,
nsBrowserApp.cpp get compiled into firefox.exe, whereas most of our code
gets compiled in xul.dll. If you define an extern variable somewhere in
firefox.exe, you cannot use it from xul.dll. If you need this kind of
cross module access, you need to export a function/variable from one
module and import and call it in the other, using MOZ_EXPORT and
MOZ_IMPORT_API and friends.
On 2014-10-24 8:36 AM, Josip Maras wrote:
Hi,
I'm extending the Firefox source in order to log some information during
web app execution, for example event, script executions, new element
creations, removals, attribute modifications, etc. Since these things occur
often, when I load a more demanding application (e.g. facebook), everything
hangs for a while, because there is a large number of relatively small
pieces of information being written to a file.
Because of this, i was thinking of doing it in batches, having one global
extern stream variable where all longs are added to, and when enough
information gets accumulated to write it to a file, and avoid these
hangings.
Since I'm a noob C++ programmer I will paste some code of what i've done,
just in case:
I have one header file with the extern variable:
-- FC_ExternStreamDeclaration.h --
#ifndef FC_EXTERN_VAR_DECL_H
#define FC_EXTERN_VAR_DECL_H
#include stdio.h
#include sstream
#include fstream
extern std::stringstream FC_LOG_STREAM;
#endif //FC_EXTERN_VAR_DECL_H
and one header with the logging functionality
-- FC_Log.h --
#pragma once
#ifndef FC_LOG_H
#define FC_LOG_H
#include stdio.h
#include sstream
#include fstream
#include FC_ExternStreamDeclaration.h
inline void WriteToStream(const char** fragments, int length)
{
//add to stream
//if stream big enough write to file
}
#endif //FC_LOG_H
and in nsBrowserApp.cpp where the main function is defined, i define the
variable and include the headers
- nsBrowserApp.cpp ---
#include FC_ExternStreamDeclaration.h
#include FC_Log.h
std::stringstream FC_LOG_STREAM;
And then, in each file from which i do logging, i include the header files
and call the WriteToStream function. For example, I'm doing it from
Interpreter.cpp, Eval.cpp, nsINode.h, etc. (all over the code).
And i thought that this will be enough. However, i get: error LNK2001
unresolved external symbol stringstream FC_LOG_STREAM. If i define it
there, the compilation succeeds, but I effectively get two different
streams, one for the JavaScript execution, and the other for the DOM and
the rest of the browser.
I'm thinking this happens because these variables are in different dlls.
I would be grateful for any tips about fixing it.
Sorry for the long question.
Thank you,
Josip
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Problem with batch logging
Thanks for the tip. Can you perhaps drop the internal stream class names, so I
can investigate a bit on how to use them.
Thank you!
On Friday, October 24, 2014 4:56:07 PM UTC+2, David Rajchenbach-Teller wrote:
Well, for one thing, never perform I/O on the main thread. It really
kills performance, as you witness. No amount of batching is going to be
sufficient.
Also, why don't you use our stream classes?
Cheers,
David
On 24/10/14 14:36, Josip Maras wrote:
Hi,
I'm extending the Firefox source in order to log some information during
web app execution, for example event, script executions, new element
creations, removals, attribute modifications, etc. Since these things occur
often, when I load a more demanding application (e.g. facebook), everything
hangs for a while, because there is a large number of relatively small
pieces of information being written to a file.
Because of this, i was thinking of doing it in batches, having one global
extern stream variable where all longs are added to, and when enough
information gets accumulated to write it to a file, and avoid these
hangings.
Since I'm a noob C++ programmer I will paste some code of what i've done,
just in case:
I have one header file with the extern variable:
-- FC_ExternStreamDeclaration.h --
#ifndef FC_EXTERN_VAR_DECL_H
#define FC_EXTERN_VAR_DECL_H
#include stdio.h
#include sstream
#include fstream
extern std::stringstream FC_LOG_STREAM;
#endif //FC_EXTERN_VAR_DECL_H
and one header with the logging functionality
-- FC_Log.h --
#pragma once
#ifndef FC_LOG_H
#define FC_LOG_H
#include stdio.h
#include sstream
#include fstream
#include FC_ExternStreamDeclaration.h
inline void WriteToStream(const char** fragments, int length)
{
//add to stream
//if stream big enough write to file
}
#endif //FC_LOG_H
and in nsBrowserApp.cpp where the main function is defined, i define the
variable and include the headers
- nsBrowserApp.cpp ---
#include FC_ExternStreamDeclaration.h
#include FC_Log.h
std::stringstream FC_LOG_STREAM;
And then, in each file from which i do logging, i include the header files
and call the WriteToStream function. For example, I'm doing it from
Interpreter.cpp, Eval.cpp, nsINode.h, etc. (all over the code).
And i thought that this will be enough. However, i get: error LNK2001
unresolved external symbol stringstream FC_LOG_STREAM. If i define it
there, the compilation succeeds, but I effectively get two different
streams, one for the JavaScript execution, and the other for the DOM and
the rest of the browser.
I'm thinking this happens because these variables are in different dlls.
I would be grateful for any tips about fixing it.
Sorry for the long question.
Thank you,
Josip
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
--
David Rajchenbach-Teller, PhD
Performance Team, Mozilla
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Problem with batch logging
On 28/10/14 09:35, Josip Maras wrote: Thanks for the tip. Can you perhaps drop the internal stream class names, so I can investigate a bit on how to use them. Thank you! You may want to open an nsAtomicFileOutputStream on a thread: http://dxr.mozilla.org/mozilla-central/source/netwerk/base/src/nsFileStreams.h?from=nsAtomicFileOutputStreamcase=true#239 Cheers, David -- David Rajchenbach-Teller, PhD Performance Team, Mozilla signature.asc Description: OpenPGP digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Problem with batch logging
On 2014-10-28 4:16 AM, Josip Maras wrote: Hi Ehsan, Yes, in my opinion that is the problem. I'm trying to use the global stream variable across module boundaries, more specifically from the following source files: content\base\src\Element.cpp; layout\style\nsCSSParser.cpp; content\base\src\nsINode.cpp, js\src\builtin\Eval.cpp, js\src\vm\Interpreter.cpp, js\src\jsfun.cpp, browser\app\nsBrowserApp.cpp, content\base\src\nsDocument.cpp, dom\base\nsGlobalWindow.cpp, parser\html\nsHtml5TreeOperation.cpp, content\html\document\src\nsHTMLContentSink.cpp So it's across at least two different modules. I've tried to follow your advice in using MOZ_EXPORT and MOZ_IMPORT, but I'm not getting anywhere :-/ The application runs, but it crashes. Of the top of your head, do you maybe know of a variable that is used in a similar way, so that I can look into how this is supposed to be done. For example, if I wanted to create a stream in nsBrowserApp.cpp and then use that stream from the other files what should I do? Do I put: extern MOZ_EXPORT std::stringstream FC_LOG_STREAM; in that file (or in the header included from that file, and then i define the varible in the .cpp file) and then use MOZ_IMPORT std::stringstream FC_LOG_STREAM; in the other files/headers or? Sorry for taking up your time, but getting into the source code of Firefox, especially since I've done very little real world development in C++ is a bit overwhelming. Any help will be appreciated! These macros just expand to __declspec(dllimport/dllexport), which is documented here: http://msdn.microsoft.com/en-us/library/3y1sfaz2.aspx Hopefully you can fix your problem following those instructions. Note that due to the nature of how we load xul.dll (which is done dynamically at runtime), I think your best bet is to export the variable from firefox.exe and import it in xul.dll, otherwise firefox.exe will not load because it will try to import a symbol from xul.dll but that DLL does not exist at startup time. We export and import some functions in the code base, but I can't think of a place where we export variables off the top of my head. grepping for the above macros should lead you in the right direction. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Problem with batch logging
On Tue, Oct 28, 2014 at 11:56:49AM -0400, Ehsan Akhgari wrote: On 2014-10-28 4:16 AM, Josip Maras wrote: Hi Ehsan, Yes, in my opinion that is the problem. I'm trying to use the global stream variable across module boundaries, more specifically from the following source files: content\base\src\Element.cpp; layout\style\nsCSSParser.cpp; content\base\src\nsINode.cpp, js\src\builtin\Eval.cpp, js\src\vm\Interpreter.cpp, js\src\jsfun.cpp, browser\app\nsBrowserApp.cpp, content\base\src\nsDocument.cpp, dom\base\nsGlobalWindow.cpp, parser\html\nsHtml5TreeOperation.cpp, content\html\document\src\nsHTMLContentSink.cpp So it's across at least two different modules. I've tried to follow your advice in using MOZ_EXPORT and MOZ_IMPORT, but I'm not getting anywhere :-/ The application runs, but it crashes. Of the top of your head, do you maybe know of a variable that is used in a similar way, so that I can look into how this is supposed to be done. For example, if I wanted to create a stream in nsBrowserApp.cpp and then use that stream from the other files what should I do? Do I put: extern MOZ_EXPORT std::stringstream FC_LOG_STREAM; in that file (or in the header included from that file, and then i define the varible in the .cpp file) and then use MOZ_IMPORT std::stringstream FC_LOG_STREAM; in the other files/headers or? Sorry for taking up your time, but getting into the source code of Firefox, especially since I've done very little real world development in C++ is a bit overwhelming. Any help will be appreciated! These macros just expand to __declspec(dllimport/dllexport), which is documented here: http://msdn.microsoft.com/en-us/library/3y1sfaz2.aspx Hopefully you can fix your problem following those instructions. Note that due to the nature of how we load xul.dll (which is done dynamically at runtime), I think your best bet is to export the variable from firefox.exe and import it in xul.dll, otherwise firefox.exe will not load because it will try to import a symbol from xul.dll but that DLL does not exist at startup time. It's not possible to use a symbol in an executable from a dll. Not directly. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Problem with batch logging
On Wed, Oct 29, 2014 at 08:08:23AM +0900, Mike Hommey wrote: On Tue, Oct 28, 2014 at 11:56:49AM -0400, Ehsan Akhgari wrote: On 2014-10-28 4:16 AM, Josip Maras wrote: Hi Ehsan, Yes, in my opinion that is the problem. I'm trying to use the global stream variable across module boundaries, more specifically from the following source files: content\base\src\Element.cpp; layout\style\nsCSSParser.cpp; content\base\src\nsINode.cpp, js\src\builtin\Eval.cpp, js\src\vm\Interpreter.cpp, js\src\jsfun.cpp, browser\app\nsBrowserApp.cpp, content\base\src\nsDocument.cpp, dom\base\nsGlobalWindow.cpp, parser\html\nsHtml5TreeOperation.cpp, content\html\document\src\nsHTMLContentSink.cpp So it's across at least two different modules. I've tried to follow your advice in using MOZ_EXPORT and MOZ_IMPORT, but I'm not getting anywhere :-/ The application runs, but it crashes. Of the top of your head, do you maybe know of a variable that is used in a similar way, so that I can look into how this is supposed to be done. For example, if I wanted to create a stream in nsBrowserApp.cpp and then use that stream from the other files what should I do? Do I put: extern MOZ_EXPORT std::stringstream FC_LOG_STREAM; in that file (or in the header included from that file, and then i define the varible in the .cpp file) and then use MOZ_IMPORT std::stringstream FC_LOG_STREAM; in the other files/headers or? Sorry for taking up your time, but getting into the source code of Firefox, especially since I've done very little real world development in C++ is a bit overwhelming. Any help will be appreciated! These macros just expand to __declspec(dllimport/dllexport), which is documented here: http://msdn.microsoft.com/en-us/library/3y1sfaz2.aspx Hopefully you can fix your problem following those instructions. Note that due to the nature of how we load xul.dll (which is done dynamically at runtime), I think your best bet is to export the variable from firefox.exe and import it in xul.dll, otherwise firefox.exe will not load because it will try to import a symbol from xul.dll but that DLL does not exist at startup time. It's not possible to use a symbol in an executable from a dll. Not directly. The best place for something that needs to be shared between firefox.exe and xul.dll is mozglue.dll. Both are linked against it. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Problem with batch logging
Well, for one thing, never perform I/O on the main thread. It really
kills performance, as you witness. No amount of batching is going to be
sufficient.
Also, why don't you use our stream classes?
Cheers,
David
On 24/10/14 14:36, Josip Maras wrote:
Hi,
I'm extending the Firefox source in order to log some information during web
app execution, for example event, script executions, new element creations,
removals, attribute modifications, etc. Since these things occur often, when
I load a more demanding application (e.g. facebook), everything hangs for a
while, because there is a large number of relatively small pieces of
information being written to a file.
Because of this, i was thinking of doing it in batches, having one global
extern stream variable where all longs are added to, and when enough
information gets accumulated to write it to a file, and avoid these hangings.
Since I'm a noob C++ programmer I will paste some code of what i've done,
just in case:
I have one header file with the extern variable:
-- FC_ExternStreamDeclaration.h --
#ifndef FC_EXTERN_VAR_DECL_H
#define FC_EXTERN_VAR_DECL_H
#include stdio.h
#include sstream
#include fstream
extern std::stringstream FC_LOG_STREAM;
#endif //FC_EXTERN_VAR_DECL_H
and one header with the logging functionality
-- FC_Log.h --
#pragma once
#ifndef FC_LOG_H
#define FC_LOG_H
#include stdio.h
#include sstream
#include fstream
#include FC_ExternStreamDeclaration.h
inline void WriteToStream(const char** fragments, int length)
{
//add to stream
//if stream big enough write to file
}
#endif //FC_LOG_H
and in nsBrowserApp.cpp where the main function is defined, i define the
variable and include the headers
- nsBrowserApp.cpp ---
#include FC_ExternStreamDeclaration.h
#include FC_Log.h
std::stringstream FC_LOG_STREAM;
And then, in each file from which i do logging, i include the header files
and call the WriteToStream function. For example, I'm doing it from
Interpreter.cpp, Eval.cpp, nsINode.h, etc. (all over the code).
And i thought that this will be enough. However, i get: error LNK2001
unresolved external symbol stringstream FC_LOG_STREAM. If i define it there,
the compilation succeeds, but I effectively get two different streams, one
for the JavaScript execution, and the other for the DOM and the rest of the
browser.
I'm thinking this happens because these variables are in different dlls.
I would be grateful for any tips about fixing it.
Sorry for the long question.
Thank you,
Josip
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
--
David Rajchenbach-Teller, PhD
Performance Team, Mozilla
signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Problem with batch logging
It's hard to determine exactly what's happening because your post
doesn't explain where you're using this variable, but note that extern
variables cannot be used across module boundaries. For example,
nsBrowserApp.cpp get compiled into firefox.exe, whereas most of our code
gets compiled in xul.dll. If you define an extern variable somewhere in
firefox.exe, you cannot use it from xul.dll. If you need this kind of
cross module access, you need to export a function/variable from one
module and import and call it in the other, using MOZ_EXPORT and
MOZ_IMPORT_API and friends.
On 2014-10-24 8:36 AM, Josip Maras wrote:
Hi,
I'm extending the Firefox source in order to log some information during web
app execution, for example event, script executions, new element creations,
removals, attribute modifications, etc. Since these things occur often, when I
load a more demanding application (e.g. facebook), everything hangs for a
while, because there is a large number of relatively small pieces of
information being written to a file.
Because of this, i was thinking of doing it in batches, having one global
extern stream variable where all longs are added to, and when enough
information gets accumulated to write it to a file, and avoid these hangings.
Since I'm a noob C++ programmer I will paste some code of what i've done,
just in case:
I have one header file with the extern variable:
-- FC_ExternStreamDeclaration.h --
#ifndef FC_EXTERN_VAR_DECL_H
#define FC_EXTERN_VAR_DECL_H
#include stdio.h
#include sstream
#include fstream
extern std::stringstream FC_LOG_STREAM;
#endif //FC_EXTERN_VAR_DECL_H
and one header with the logging functionality
-- FC_Log.h --
#pragma once
#ifndef FC_LOG_H
#define FC_LOG_H
#include stdio.h
#include sstream
#include fstream
#include FC_ExternStreamDeclaration.h
inline void WriteToStream(const char** fragments, int length)
{
//add to stream
//if stream big enough write to file
}
#endif //FC_LOG_H
and in nsBrowserApp.cpp where the main function is defined, i define the
variable and include the headers
- nsBrowserApp.cpp ---
#include FC_ExternStreamDeclaration.h
#include FC_Log.h
std::stringstream FC_LOG_STREAM;
And then, in each file from which i do logging, i include the header files and
call the WriteToStream function. For example, I'm doing it from
Interpreter.cpp, Eval.cpp, nsINode.h, etc. (all over the code).
And i thought that this will be enough. However, i get: error LNK2001
unresolved external symbol stringstream FC_LOG_STREAM. If i define it there,
the compilation succeeds, but I effectively get two different streams, one for
the JavaScript execution, and the other for the DOM and the rest of the browser.
I'm thinking this happens because these variables are in different dlls.
I would be grateful for any tips about fixing it.
Sorry for the long question.
Thank you,
Josip
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
